RE: Deep concerns about the future of EPUB

Hi Daniel,

As I see things,  you are correct on a very important point. In hindsight, the EPUB WG decision to label the final release of EPUB under IDPF auspices "3.1" was a serious mistake, due primarily to behavior of EPUBCheck-based validation software that could not accept 3.1 files, and urgent action is necessary to mitigate confusion and potential for forking. Even had we, as anticipated at the time, timely updated EPUBCheck to 3.1, there would likely still been problems in adoption due to the many different deployed instances of EBPUCheck that could not or would not have been updated immediately. And since EPUBCheck upgrade was delayed, it's of course been even worse.

There is a plan being formulated that has begun being discussed in the EPUB 3 CG and with the chairs of the other Publishing@W3C groups, and W3C Team contacts, to harmonize things by the CG publishing what will be (at least from EPUB package version perspective) an EPUB 3.0.2 that will in effect supersede EPUB 3.1 and maintain practical compatibility among all  EPUB 3.0.x versions (with other changes TBD). More on this very soon and I hope that you will get involved constructively in the EPUB 3 CG in this work.

A year ago, the IDPF EPUB WG and leadership was rushing to complete EPUB 3.1 as one of the final steps before completing the combination with W3C. I regret that in the time crunch we didn't forsee the future better in regards to versioning. I personally pushed the IDPF WG leadership re: calling the new version EPUB 3.0.2 but ultimately I went along with the final decision so I'm as culpable as anyone else.

That being said I think you're engaging in a logical fallacy by claiming that a single mistake by the IDPF EPUB WG over a year ago implies that we must now radically rethink the way W3C Publishing@W3C groups are structured in general, as well as how ongoing development of EPUB 3 will be handled specifically.

EPUB 3.0(.1) has been "good enough" for most purposes so the lack of adoption of EPUB 3.1 in the last 10 months is, while not ideal, not in fact a major crisis. Zero publications are going un-published for lack of EPUB 3.1 feature support, so far as I am aware. 

And that the EPUB 3 CG is beginning work to correct the version Catch-22 and set the stage for future development of EPUB 3.0.x in an orderly and reliable manner  is to me a datapoint that we have a reasonable process and structure in place already. The proof will be in the pudding: either the EPUB 3 CG will develop in reasonably short order a solution that fixes things and quickly gets on an adoption vector, in which case we'll have clear validation that things are OK, or it won't, in which case we should then look at more radical fixes. I hope that even though you may have reservations about the CG process for EPUB development you will pitch in and help. At the end of the day I feel that processes matter less than who comes to the table to make things work, and your contributions to that are very valuable.

And, at the end of the day the grass is not necessarily greener on the other side. .  EPUB 3.0.x is certainly not a perfect specification, nor is its adoption 100% consistent, but W3C processes and W3C specificaitons have their issues and critics too (and I believe that sometimes includes you). And the key reasons we didn't move EPUB 3.x into a WG to begin with - that we have zero evidence that a W3C WG is suitable for incremental evolution of an existing and adopted specification that was not originally developed under W3C processes and IP policy, and that we want that incremental evolution of EPUB 3.x to be able to move ahead nimbly at the pace dictated by industry needs, not at the more stately pace of a WG - still hold true.

Thanks,

--Bill


-----Original Message-----
From: Daniel Glazman [mailto:daniel.glazman@disruptive-innovations.com] 
Sent: Tuesday, December 26, 2017 3:06 AM
To: public-publishingbg@w3.org
Cc: Bill McCoy <bmccoy@w3.org>; Jeff Jaffe <jeff@w3.org>; Tim Berners-Lee <timbl@w3.org>; eb2m-mrt@asahi-net.or.jp
Subject: Deep concerns about the future of EPUB


+----------------------------------------------------------------+
|TL;DR: the future of EPUB is a fiasco of XHTML2 magnitude.      |
|       Urgent actions required.                                 |
+----------------------------------------------------------------+


During the discussions about EPUB 3.1 and its adoption process, two sets of comments were ignored, or at best severely underestimated:

  - yours truly said the industry was barely starting to convert the
    masses of EPUB 2.x to incompatible 3.0.1 and releasing so quickly
    yet another incompatible 3.1 was a strategic mistake. As an
    IMPLEMENTOR, and probably the only implementor in the world
    releasing EPUB 3.1 in a content editor, I had a lot of reservations
    about the spec despite of its simplifications (in particular in
    its metadata section; the simplifications were technically ok, but
    overkill business-wise). Implementing 3.1 implied yet another
    parser and yet another package processor. It was then
    clear the cost for production chains would not be negligible.

  - the Japanese publishing industry sent a very clear and very
    important letter [1] at the same moment, back in february 2016,
    stating that 3.1 was a no-go for them because of the cost the
    incompatibilities with 3.0.1 would imply. Murata-san recently
    contributed to github issue #982 [2] more arguments, again all very
    clear and, I must say, perfectly understandable.

The adoption process continued on, despite of the warnings, at fast pace. The speed of adoption of 3.1, added to the lack of implementation report based on conformant implementations, was also pointed out as a big issue. No change was implemented, despite of the plausible short- term merger with W3C.

Less than two years later, the above problems are now blockers:

  - the Japanese industry, the largest 3.0.x adopter in the world,
    clearly rejects adoption of 3.1, with good reasons.

  - if the EPUB ecosystem is anemic because of its complexity, the
    3.1 ecosystem is, as predicted, infinitesimal. Almost no software
    chain widely available in the wild implements it. As far as I
    can tell, less than 1% of BlueGriffon EPUB users create and edit
    3.1. More than 99% use it for 2.x and 3.0.1.

  - the simplifications of 3.1 over 3.0.1 were clearly not enough to
    justify a new version of the spec - and the migration costs - so
    quickly.

Despite of the above, the Publishing activity at W3C recreated more or less the working habits of IDPF inside W3C. We are already working on a successor to EPUB 3 that will for sure suffer from the same ailments. That work is done inside a Working Group but there are extremely diverging opinions about what should be the next version of EPUB. Dissonant voices are, for the same reasons and the same way as before, neglected and the work is done at extremely fast pace, in a process that I call suboptimal. Backward-compatibility has never been a first-class element in the standardization process despite of being a notable highlight in the specs. I have always said, with my implementor's hat on, that the backward-compatibility of the EPUB spec ecosystem just does not exist. I can technically prove this.

EPUB3 (hear 3.1) is maintained by a Community Group. We now see that the gordian knot of EPUB is not v4 but v3. v3 is where the biggest effort matters the most. The whole architecture of the Publishing activity at W3C, with its CG, WG, BG and Steering Committee, is not tailored to meet the Industry's needs. The BG and the Steering Committee, « ensuring that the interests of the publishing industry are appropriately reflected in such publishing activity work », are the children of the committee that pushed 3.1 so fast despite of the aforementioned warnings. The BG is in effect leading the CG and took "a lead role in the process of developing (the) charter" of the WG. We now see that the whole thing is not in line, at all, with the largest EPUB industry in the world.

In that light, merging with W3C was seen, at least by me, as a way of preventing such catastrophic errors to happen again. Formal Objections and the whole W3C Process could vastly help. The creation of the BG and its clear handhold on the standardization groups of the Publishing domain at W3C, were a signal in the other direction. The creation of the Groups was again done at fast pace, in a suboptimal process, only to make sure the Groups were created before a Publishing Summit.

The result is a incredible capharnaüm:

  - the last version of EPUB sees no adoption. Its ecosystem is
    virtually non-existent. Seen from here, EPUB is a dead end.

  - each version of EPUB is incompatible with others. Again, I can
    technically prove this.

  - the technical level of the specifications is, and I say this
    with a W3C dinosaur's eye, low. EPUB is a weak spec that is not
    meeting the quality and implementation constraints levels of other
    Standard Bodies.

  - ISO/IEC adoption of 3.1 would make 3.0.x obsolete

  - Publishing@W3C is not meeting Industry's needs

  - the WG deals with the future of EPUB while the CG maintains the
    existing line of EPUB. And the top of the existing line is now
    rejected by one of our largest user. We now see the most important
    bit here is not the future but the present. As said before, the EPUB
    specs line changes FAR TOO FAST for the Industry. As a result, the
    architecture of the Publishing activity is wrong: the WG should deal
    with 3.x, allowing the full power of W3C Process to apply, while the
    CG should deal with EPUB 4 and WP that are still a topic of complex
    and totally unstabilized discussions.

  - I hope it's now totally out of question to do work in the Publishing
    activity without serious exit criteria based on implementation
    reports. And when I say "implementations", I mean software delivered
    to masses passing a test suite, not prototypes of limited spread or
    lists of users... For the time being, the exit criteria from
    Publishing@W3C are clearly not at W3C level.

As the sole implementor of 3.1 in an editing environment, I am then asking the Publishing BG to urgently clarify the situation through a deep self-criticism loop. Implementations of EPUB are EXTREMELY expensive and it's now out of question to continue moving on without a clear view, based on an thorough analysis of requirements and facts, of where we're coming from and where we're going to. It's pointless to keep discussing 3.2, 4, WP until this is done because the future is, for the time being, totally uncertain. The architecture of the Publishing activity at W3C should be reviewed, and modified. The goals of that Publishing activity, namely EPUB 4, should be reviewed and probably deferred. WP, and their disruptive changes to the technical aspects of our industry, are probably not what the market needs right now.

We don't need a new Publishing Summit. We need a Future of EPUB face-to-face meeting, with 3 days of open intense discussions, no panels, no conference, no blah-blah and I think we need it as soon as possible. We need consensus, good handling of objections, W3M supervision and strict W3C Process.

Last but not least, the first mission of the self-criticism loop requested above should be to make sure such a mess never happens again. As a software vendor, I am shocked by the current situation and what it potentially implies on the future evolutions of our software and the finances behind that. In particular, the letter from the Japanese publishing industry should have reverberated, back in february 2016, as a formal showstopper.

We cannot afford another fiasco of that magnitude, and we need to fix the consequences of this one:

  - why is 3.1 such a failure?
  - why didn't the understandable warnings above serve as showstoppers?
  - conclusions on IDPF standardization Process?
  - what is the next level of EPUB and when is it relevant to
    publish it, industry-wise? EPUB 3.0 was released in October 2011
    and more than six years later, the Industry is NOT ready to move
    to the next version, 3.1, that is almost not implemented at all;
    so does it make any sense to start work on another version?
  - what are the cost metrics for an EPUB update?
  - what's the correct host for such a work?
  - since backward-compatibility's requirements have grown with the
    market, is it still possible to make EPUB evolve?
  - why is EPUB making any kind of recommendation on content documents?
  - how can we avoid another hiatus of that kind? Should the Charters
    be amended?
  - is the Publishing@W3C architecture efficient?
  - what's the impact on Chairing?
  - how do we define workable exit criteria for specs that do represent
    interoperable implementability and ensure market openness?
  - EPUB 3.0.1 is NOT in-scope in the CG's Charter
  - etc.

[1]
https://docs.google.com/document/d/1LDmzu7HkFLqXqj-pO5Yx_CVVfEXGz8u81cf1pyCS7gk
[2] https://github.com/w3c/publ-epub-revision/pull/982

</Daniel>

Received on Tuesday, 26 December 2017 18:03:14 UTC