Re: Deep concerns about the future of EPUB

I'm not well-qualified to comment on the debate about technical standards,
but the test-suite discussion has circled around much of my first year at
BISG. Earlier this year, I wrote and shared the attached outline with the
publishing business group, which had asked me to come to a meeting to talk
about epubtest and moving it ahead.

The TL;DR version: to move ahead, we need to decide what we want to
accomplish with epubtest. I present a few options, but none of them stuck
with the PBG. It sounds like this debate is settling in a similar place.

On Fri, Dec 29, 2017 at 12:46 PM, Matt Garrish <matt.garrish@gmail.com>
wrote:

> > But, IMO, the WG needs a more complete set of tests if we are going to
> promulgate new versions of the spec and certify that there are two
> complying
> implementations.
>
> Absolutely agree. The "test suite" has a bit of a schizophrenic existence
> and this question of its purpose seems to come up on both sides
> (comprehensiveness and feature support). As someone who contributed to the
> original tests, I can state our objective was only to show publishers that
> content would render and be an improvement on EPUB 2.
>
> I wouldn't feel bound to making it work for any other purpose than it was
> originally conceived. I have no doubt there are better approaches for a
> comprehensive test suite, and whether it even meets the needs of showing
> epub feature support can also be questioned. I don't see it as a bad thing
> to retire it entirely if it's run its useful life, either.
>
> Matt
>
> > -----Original Message-----
> > From: Ric Wright [mailto:rkwright@geofx.com]
> > Sent: December 29, 2017 12:22 PM
> > To: Matt Garrish <matt.garrish@gmail.com>; 'Daniel Glazman'
> > <daniel.glazman@disruptive-innovations.com>; 'Bill McCoy'
> > <bmccoy@w3.org>; public-publishingbg@w3.org
> > Cc: 'Jeff Jaffe' <jeff@w3.org>; 'Tim Berners-Lee' <timbl@w3.org>; eb2m-
> > mrt@asahi-net.or.jp
> > Subject: Re: Deep concerns about the future of EPUB
> >
> > Matt,
> >
> > I completely agree with this. The problem I see is TS *has* become more
> of
> a
> > can-I-use and simple unit-tests, so it is very useful as a set of
> regression tests
> > (which is how Readium uses it for the most part).  But, IMO, the WG needs
> a
> > more complete set of tests if we are going to promulgate new versions of
> the
> > spec and certify that there are two complying implementations.
> >
> > So what DO we need? It would be an excellent exercise to outline in some
> > detail what DOES need to be tested and how and what the success criteria
> > would consist of. Let me (and hopefully others) give that some thought.
> >
> > Thanks
> > Ric
> >
> >
> > On 12/28/17, 1:08 PM, "Matt Garrish" <matt.garrish@gmail.com> wrote:
> >
> > >> Again, to be clear, the TS is an awesome batch of work, it just isn¹t
> > >enough.
> > >
> > >It never really had aspirations to be the thing you want it to be, too..
> > >I don't recall any reading system developers were even heavily involved
> > >in putting it together. Its ambition was to test some essential
> > >rendering issues that were poorly supported in EPUB 2 or new to EPUB 3
> > >to kickstart and improve support in the early days. It then morphed
> > >with the BISG support grid, further taking it into the territory of a
> > >caniuse-like input.
> > >
> > >Matt
> > >
> > >> -----Original Message-----
> > >> From: Ric Wright [mailto:rkwright@geofx.com]
> > >> Sent: December 28, 2017 10:26 AM
> > >> To: Daniel Glazman <daniel.glazman@disruptive-innovations.com>; Bill
> > >> McCoy <bmccoy@w3.org>; public-publishingbg@w3.org
> > >> Cc: 'Jeff Jaffe' <jeff@w3.org>; 'Tim Berners-Lee' <timbl@w3.org>;
> > >> eb2m- mrt@asahi-net.or.jp
> > >> Subject: Re: Deep concerns about the future of EPUB
> > >>
> > >> I have been following this thread with great interest (Daniel, thanks
> > >>for  raising this).  There are a lot of good thoughts and arguments on
> > >>all
> > >fronts.
> > >>
> > >> That said, I tend to agree with many of the points Daniel has made
> here.
> > >> My perspective is from that of a Reading System developer/maintainer..
> > >> There are two key points I would make:
> > >>
> > >> Support for all the varieties of EPUB:
> > >> A point I have raised a number of times is that adding new or changed
> > >>features to EPUB often sounds like a good idea, but as a RS, we *HAVE*
> > >>to  support all variants and features from 2.0 to the latest - now and
> > >forever. So
> > >> incompatible or changed features are hugely problematic. At this
> > >>point,  Readium supports virtually all of 3.0.1 but we haven¹t even
> > >>considered any
> > >of
> > >> the new features in 3.1 (we¹re short-handed anyway, but even if we
> > >> weren¹t...).  The net-net is that I, as director of Readium, want to
> > >> see
> > >what
> > >> the value-added is of new features (and that they are being used)
> > >> before
> > >we
> > >> invest the time and effort to support them.
> > >>
> > >> Validation of Existing Implementations:
> > >> At this point, Readium supports around 90% of the features in 3.0.1
> > >> as measured by the EPUB TestSuite. But that being said, this isn¹t
> > >> really a
> > >very
> > >> good metric.  The EPUB TestSuite is an excellent tool as far as it
> > >> goes,
> > >but it is
> > >> significantly flawed in several ways. Here are three of
> > >> them:
> > >> 1. Almost 40% of the features it tests are really browser-level
> > >functionality.
> > >> So it is not really testing the Reading System, but the browser
> > >>engine the
> > >RS is
> > >> built on top of (I think RMSDK was the last RS that built its own
> > >>engine).
> > >This
> > >> is important, of course, but a little misleading as to what is being
> > >tested and
> > >> how thoroughly.
> > >> 2. The features are tested in a very atomic way, one isolated feature
> > >>at a  time. This is understandable and the right way to start, but
> > >>that vast
> > >majority
> > >> of bugs and issues Readium encounters are more complex interactions,
> > >> or odd variants of features.  I cannot offhand recall a single bug we
> > >> have encountered lately that was found through, or could be tested
> > >> with, the EPUB TestSuite.  To be clear, the TS is very valuable, it
> > >> simply doesn¹t
> > >go
> > >> anywhere near far enough.
> > >> 3. There are few if any failure cases.  Or stress cases. The TS is so
> > >constructed
> > >> such that the tests are clean and well-behaved.  If only the content
> > >>we  received from all and sundry were equally well-behaved. :-) The TS
> > >>should  also be testing the RS to see how they handle errors.  In some
> > >>cases, the  EPUB spec does in fact specify how errors or bad and/or
> > >>illegal content  should be handled, but the TS doesn¹t test these. One
> > >>small example:
> > >>For
> > >> reasons none of us who were there can recall, declarative SVG
> > >>animation is  not allowed in EPUB files.  EPUBCheck dutifully
> > >>complains about
> > >(declarative)
> > >> SVG animation, but the TS doesn¹t test that such content is ignored.
> > >>The
> > >> overhead to disallow such content in a RS is quite large, so Readium
> > >simply
> > >> doesn¹t bother and SVG animation works fine even though it should be
> > >>rejected.  Again, to be clear, the TS is an awesome batch of work, it
> > >>just
> > >isn¹t
> > >> enough.
> > >>
> > >> My two cents, FWIW.
> > >>
> > >> Ric
> > >>
> > >>
> > >>
> > >> On 12/27/17, 4:23 AM, "Daniel Glazman"
> > >> <daniel.glazman@disruptive-innovations.com> wrote:
> > >>
> > >> >Le 26/12/2017 à 19:02, Bill McCoy a écrit :
> > >> >
> > >> >> That being said I think you're engaging in a logical fallacy by
> > >> >>claiming that a single mistake by the IDPF EPUB WG over a year ago
> > >> >>implies that we must now radically rethink the way W3C
> > >> >>Publishing@W3C groups are structured in general, as well as how
> > >> >>ongoing development of EPUB 3 will be handled specifically.
> > >> >
> > >> >I respectfully disagree 100%. If the market is already unable to
> > >> >digest
> > >> >3.1 six years after the release of 3.0, when will it be able to
> > >> >digest 4.0, Web Publications, Portable Web Publications and more?
> > >> >Given the EPUB 3 situation, any investment now on WP, PWP, EPUB4 is
> > >> >too early and too expensive. The EPUB market is just unable to
> > >> >absorb a new spec every three years. Our whole model is flawed.
> > >> >
> > >> >I suggest we take that chartered time to make a 3.x version conform
> > >> >to regular W3C exit criteria, with a test suite and real
> > >> >implementation reports, and reach W3C REC (instead of "Recommended
> > >> >Specification" IDPF
> > >> >designation) instead of diving into useless expensive blue-sky
> dreams.
> > >> >
> > >> >*THAT* would be *immensely* useful to *everyone*.
> > >> >
> > >> >Let me repeat myself on one important point: backwards-compatibility
> > >> >requirements of EPUB increase as the EPUB market increase. They will
> > >> >become mandatory soon. An ebook is an ebook is an ebook. Nobody
> > >> >wants to maintain several technical versions of the same ebook for a
> > >> >given technology, nobody wants to update (at a cost) ebooks
> > >> >published eons ago that should just work, nobody wants to leave
> > >> >legacy reading systems unattended because it could kill the
> > >> >ecosystem, and we're not even able to get rid of our ugliest design
> > >> >choices any more. Hence my
> > >> >conclusion: EPUB is a dead end. A technological disruption will
> > >> >happen, it's not a question of "if" any more but only a question of
> > "when".
> > >> >
> > >> ></Daniel>
> > >> >
> > >>
> > >>
> > >
> > >
> >
>
>
>
>


-- 
Brian F. O'Leary
Executive Director, Book Industry Study Group
232 Madison Avenue, Suite 1400
New York, NY 10016

(646) 336-7141 office
(973) 985-9880 mobile