Deep concerns about the future of EPUB

+----------------------------------------------------------------+
|TL;DR: the future of EPUB is a fiasco of XHTML2 magnitude.      |
|       Urgent actions required.                                 |
+----------------------------------------------------------------+


During the discussions about EPUB 3.1 and its adoption process, two sets
of comments were ignored, or at best severely underestimated:

  - yours truly said the industry was barely starting to convert the
    masses of EPUB 2.x to incompatible 3.0.1 and releasing so quickly
    yet another incompatible 3.1 was a strategic mistake. As an
    IMPLEMENTOR, and probably the only implementor in the world
    releasing EPUB 3.1 in a content editor, I had a lot of reservations
    about the spec despite of its simplifications (in particular in
    its metadata section; the simplifications were technically ok, but
    overkill business-wise). Implementing 3.1 implied yet another
    parser and yet another package processor. It was then
    clear the cost for production chains would not be negligible.

  - the Japanese publishing industry sent a very clear and very
    important letter [1] at the same moment, back in february 2016,
    stating that 3.1 was a no-go for them because of the cost the
    incompatibilities with 3.0.1 would imply. Murata-san recently
    contributed to github issue #982 [2] more arguments, again all very
    clear and, I must say, perfectly understandable.

The adoption process continued on, despite of the warnings, at fast
pace. The speed of adoption of 3.1, added to the lack of implementation
report based on conformant implementations, was also pointed out as a
big issue. No change was implemented, despite of the plausible short-
term merger with W3C.

Less than two years later, the above problems are now blockers:

  - the Japanese industry, the largest 3.0.x adopter in the world,
    clearly rejects adoption of 3.1, with good reasons.

  - if the EPUB ecosystem is anemic because of its complexity, the
    3.1 ecosystem is, as predicted, infinitesimal. Almost no software
    chain widely available in the wild implements it. As far as I
    can tell, less than 1% of BlueGriffon EPUB users create and edit
    3.1. More than 99% use it for 2.x and 3.0.1.

  - the simplifications of 3.1 over 3.0.1 were clearly not enough to
    justify a new version of the spec - and the migration costs - so
    quickly.

Despite of the above, the Publishing activity at W3C recreated more or
less the working habits of IDPF inside W3C. We are already working on
a successor to EPUB 3 that will for sure suffer from the same
ailments. That work is done inside a Working Group but there are
extremely diverging opinions about what should be the next version of
EPUB. Dissonant voices are, for the same reasons and the same way as
before, neglected and the work is done at extremely fast pace, in a
process that I call suboptimal. Backward-compatibility has never been
a first-class element in the standardization process despite of being
a notable highlight in the specs. I have always said, with my
implementor's hat on, that the backward-compatibility of
the EPUB spec ecosystem just does not exist. I can technically prove
this.

EPUB3 (hear 3.1) is maintained by a Community Group. We now see that
the gordian knot of EPUB is not v4 but v3. v3 is where the biggest
effort matters the most. The whole architecture of the Publishing
activity at W3C, with its CG, WG, BG and Steering Committee, is not
tailored to meet the Industry's needs. The BG and the Steering
Committee, « ensuring that the interests of the publishing industry are
appropriately reflected in such publishing activity work », are the
children of the committee that pushed 3.1 so fast despite of the
aforementioned warnings. The BG is in effect leading the CG and
took "a lead role in the process of developing (the) charter" of the
WG. We now see that the whole thing is not in line, at all, with the
largest EPUB industry in the world.

In that light, merging with W3C was seen, at least by me, as a way
of preventing such catastrophic errors to happen again. Formal
Objections and the whole W3C Process could vastly help. The creation of
the BG and its clear handhold on the standardization groups of the
Publishing domain at W3C, were a signal in the other direction. The
creation of the Groups was again done at fast pace, in a suboptimal
process, only to make sure the Groups were created before a Publishing
Summit.

The result is a incredible capharnaüm:

  - the last version of EPUB sees no adoption. Its ecosystem is
    virtually non-existent. Seen from here, EPUB is a dead end.

  - each version of EPUB is incompatible with others. Again, I can
    technically prove this.

  - the technical level of the specifications is, and I say this
    with a W3C dinosaur's eye, low. EPUB is a weak spec that is not
    meeting the quality and implementation constraints levels of other
    Standard Bodies.

  - ISO/IEC adoption of 3.1 would make 3.0.x obsolete

  - Publishing@W3C is not meeting Industry's needs

  - the WG deals with the future of EPUB while the CG maintains the
    existing line of EPUB. And the top of the existing line is now
    rejected by one of our largest user. We now see the most important
    bit here is not the future but the present. As said before, the EPUB
    specs line changes FAR TOO FAST for the Industry. As a result, the
    architecture of the Publishing activity is wrong: the WG should deal
    with 3.x, allowing the full power of W3C Process to apply, while the
    CG should deal with EPUB 4 and WP that are still a topic of complex
    and totally unstabilized discussions.

  - I hope it's now totally out of question to do work in the Publishing
    activity without serious exit criteria based on implementation
    reports. And when I say "implementations", I mean software delivered
    to masses passing a test suite, not prototypes of limited spread or
    lists of users... For the time being, the exit criteria from
    Publishing@W3C are clearly not at W3C level.

As the sole implementor of 3.1 in an editing environment, I am then
asking the Publishing BG to urgently clarify the situation through a
deep self-criticism loop. Implementations of EPUB are EXTREMELY
expensive and it's now out of question to continue moving on without
a clear view, based on an thorough analysis of requirements and facts,
of where we're coming from and where we're going to. It's pointless
to keep discussing 3.2, 4, WP until this is done because the future
is, for the time being, totally uncertain. The architecture of the
Publishing activity at W3C should be reviewed, and modified. The goals
of that Publishing activity, namely EPUB 4, should be reviewed and
probably deferred. WP, and their disruptive changes to the technical
aspects of our industry, are probably not what the market needs right
now.

We don't need a new Publishing Summit. We need a Future of EPUB
face-to-face meeting, with 3 days of open intense discussions, no
panels, no conference, no blah-blah and I think we need it as soon as
possible. We need consensus, good handling of objections, W3M
supervision and strict W3C Process.

Last but not least, the first mission of the self-criticism loop
requested above should be to make sure such a mess never happens
again. As a software vendor, I am shocked by the current situation
and what it potentially implies on the future evolutions of our
software and the finances behind that. In particular, the letter from
the Japanese publishing industry should have reverberated, back in
february 2016, as a formal showstopper.

We cannot afford another fiasco of that magnitude, and we need to fix
the consequences of this one:

  - why is 3.1 such a failure?
  - why didn't the understandable warnings above serve as showstoppers?
  - conclusions on IDPF standardization Process?
  - what is the next level of EPUB and when is it relevant to
    publish it, industry-wise? EPUB 3.0 was released in October 2011
    and more than six years later, the Industry is NOT ready to move
    to the next version, 3.1, that is almost not implemented at all;
    so does it make any sense to start work on another version?
  - what are the cost metrics for an EPUB update?
  - what's the correct host for such a work?
  - since backward-compatibility's requirements have grown with the
    market, is it still possible to make EPUB evolve?
  - why is EPUB making any kind of recommendation on content documents?
  - how can we avoid another hiatus of that kind? Should the Charters
    be amended?
  - is the Publishing@W3C architecture efficient?
  - what's the impact on Chairing?
  - how do we define workable exit criteria for specs that do represent
    interoperable implementability and ensure market openness?
  - EPUB 3.0.1 is NOT in-scope in the CG's Charter
  - etc.

[1]
https://docs.google.com/document/d/1LDmzu7HkFLqXqj-pO5Yx_CVVfEXGz8u81cf1pyCS7gk
[2] https://github.com/w3c/publ-epub-revision/pull/982

</Daniel>

Received on Tuesday, 26 December 2017 11:06:09 UTC