RE: Deep concerns about the future of EPUB

WRT Matt's point about the test suite's schizophrenic existence, the two fundamental poles/purposes/personalities are these:

1. To inform publishers about what features of EPUB will "just work" and work properly in a given RS.

2. To enable RS developers to test whether their software works properly for content that conforms to the EPUB spec.

The historical perspective, best articulated by Brian O'Leary's summary for the BG, can be summarized like this:

--Purpose 1 was the original intention of the test suite / EPUB Grid.

--When it was implemented, it was discovered that Purpose 2 was also an important benefit/byproduct.

As time has gone on (and as Brian documents):

--It has become increasingly difficult to maintain Purpose 1, not only because of the difficulty of recruiting volunteers (which could be addressed by changing the top-down model and moving more to a caniuse/crowdsourced model) but precisely _because of_ the now rich and diverse implementation environment. What is difficult, and what may be impossible, is to keep up with all the places EPUB is implemented, including changes in OS's and browser engines on which those implementations are based. The Accessibility group, which has a very particular focus, has admirably addressed this complexity (including the added layer of AT), but it may now be the case that the original vision of the TS/EPUB Grid--to inform publishers about what features of EPUB will "just work" and work properly in a given RS (Purpose 1) in anything like a current and comprehensive way--is not just difficult but impossible for what we have called the "mainstream testing." (I don't think anybody disagrees that the accessibility testing is working and should keep going full steam ahead.)

--One corollary of Purpose 1--and this was really the main motivator for the creation of the Grid in the first place--was to demonstrate that EPUB _actually was_ being implemented in reading systems, maybe not every feature, but most of the features that publishers most needed. That was because publishers were holding back due to the perception that "EPUB 3 doesn't work anywhere." That perception is no longer a problem. We don't have a problem with EPUB 3 not being perceived as viable; it's widely implemented. (Not EPUB 3.1, of course; I mean EPUB 3 in general, which currently means EPUB 3.0.x.)

So it may be arguable that Purpose 1 is no longer needed, or that the effort to address Purpose 1 is no longer justifiable.

Instead, it appears that a focus on Purpose 2 is what is needed. We no longer need to convince the publishing industry that EPUB 3 is viable; that's pretty much a settled issue. But developers _really need_ the ability to test their software effectively. 

So I suggest that a group of _publishers_ getting together to work on this--which was what BISG did, essentially--isn't what is needed. What is needed is a group of _developers_ getting together to solve this.

--Bill K

P.S. A sidenote: both publishers and developers need _epubcheck_ to work. I'm responding to the epubtest/EPUB Grid discussion, but I'm aware that the epubcheck issue is the most urgent.

Bill Kasdorf
VP and Principal Consultant | Apex CoVantage
p:
734-904-6252  m:   734-904-6252
ISNI: http://isni.org/isni/0000000116490786
ORCiD: https://orcid.org/0000-0001-7002-4786



-----Original Message-----
From: Matt Garrish [mailto:matt.garrish@gmail.com] 
Sent: Friday, December 29, 2017 12:46 PM
To: 'Ric Wright'; 'Daniel Glazman'; 'Bill McCoy'; public-publishingbg@w3.org
Cc: 'Jeff Jaffe'; 'Tim Berners-Lee'; eb2m-mrt@asahi-net.or.jp
Subject: RE: Deep concerns about the future of EPUB

> But, IMO, the WG needs a more complete set of tests if we are going to
promulgate new versions of the spec and certify that there are two complying implementations.

Absolutely agree. The "test suite" has a bit of a schizophrenic existence and this question of its purpose seems to come up on both sides (comprehensiveness and feature support). As someone who contributed to the original tests, I can state our objective was only to show publishers that content would render and be an improvement on EPUB 2.

I wouldn't feel bound to making it work for any other purpose than it was originally conceived. I have no doubt there are better approaches for a comprehensive test suite, and whether it even meets the needs of showing epub feature support can also be questioned. I don't see it as a bad thing to retire it entirely if it's run its useful life, either.

Matt

> -----Original Message-----
> From: Ric Wright [mailto:rkwright@geofx.com]
> Sent: December 29, 2017 12:22 PM
> To: Matt Garrish <matt.garrish@gmail.com>; 'Daniel Glazman'
> <daniel.glazman@disruptive-innovations.com>; 'Bill McCoy'
> <bmccoy@w3.org>; public-publishingbg@w3.org
> Cc: 'Jeff Jaffe' <jeff@w3.org>; 'Tim Berners-Lee' <timbl@w3.org>; 
> eb2m- mrt@asahi-net.or.jp
> Subject: Re: Deep concerns about the future of EPUB
> 
> Matt,
> 
> I completely agree with this. The problem I see is TS *has* become 
> more of
a
> can-I-use and simple unit-tests, so it is very useful as a set of
regression tests
> (which is how Readium uses it for the most part).  But, IMO, the WG 
> needs
a
> more complete set of tests if we are going to promulgate new versions 
> of
the
> spec and certify that there are two complying implementations.
> 
> So what DO we need? It would be an excellent exercise to outline in 
> some detail what DOES need to be tested and how and what the success 
> criteria would consist of. Let me (and hopefully others) give that some thought.
> 
> Thanks
> Ric
> 
> 
> On 12/28/17, 1:08 PM, "Matt Garrish" <matt.garrish@gmail.com> wrote:
> 
> >> Again, to be clear, the TS is an awesome batch of work, it just 
> >> isnıt
> >enough.
> >
> >It never really had aspirations to be the thing you want it to be, too.
> >I don't recall any reading system developers were even heavily 
> >involved in putting it together. Its ambition was to test some 
> >essential rendering issues that were poorly supported in EPUB 2 or 
> >new to EPUB 3 to kickstart and improve support in the early days. It 
> >then morphed with the BISG support grid, further taking it into the 
> >territory of a caniuse-like input.
> >
> >Matt
> >
> >> -----Original Message-----
> >> From: Ric Wright [mailto:rkwright@geofx.com]
> >> Sent: December 28, 2017 10:26 AM
> >> To: Daniel Glazman <daniel.glazman@disruptive-innovations.com>; 
> >> Bill McCoy <bmccoy@w3.org>; public-publishingbg@w3.org
> >> Cc: 'Jeff Jaffe' <jeff@w3.org>; 'Tim Berners-Lee' <timbl@w3.org>;
> >> eb2m- mrt@asahi-net.or.jp
> >> Subject: Re: Deep concerns about the future of EPUB
> >>
> >> I have been following this thread with great interest (Daniel, 
> >>thanks for  raising this).  There are a lot of good thoughts and 
> >>arguments on all
> >fronts.
> >>
> >> That said, I tend to agree with many of the points Daniel has made
here.
> >> My perspective is from that of a Reading System developer/maintainer.
> >> There are two key points I would make:
> >>
> >> Support for all the varieties of EPUB:
> >> A point I have raised a number of times is that adding new or 
> >>changed features to EPUB often sounds like a good idea, but as a RS, 
> >>we *HAVE* to  support all variants and features from 2.0 to the 
> >>latest - now and
> >forever. So
> >> incompatible or changed features are hugely problematic. At this 
> >>point,  Readium supports virtually all of 3.0.1 but we havenıt even 
> >>considered any
> >of
> >> the new features in 3.1 (weıre short-handed anyway, but even if we 
> >> werenıt...).  The net-net is that I, as director of Readium, want 
> >> to see
> >what
> >> the value-added is of new features (and that they are being used) 
> >> before
> >we
> >> invest the time and effort to support them.
> >>
> >> Validation of Existing Implementations:
> >> At this point, Readium supports around 90% of the features in 3.0.1 
> >> as measured by the EPUB TestSuite. But that being said, this isnıt 
> >> really a
> >very
> >> good metric.  The EPUB TestSuite is an excellent tool as far as it 
> >> goes,
> >but it is
> >> significantly flawed in several ways. Here are three of
> >> them:
> >> 1. Almost 40% of the features it tests are really browser-level
> >functionality.
> >> So it is not really testing the Reading System, but the browser 
> >>engine the
> >RS is
> >> built on top of (I think RMSDK was the last RS that built its own 
> >>engine).
> >This
> >> is important, of course, but a little misleading as to what is 
> >> being
> >tested and
> >> how thoroughly.
> >> 2. The features are tested in a very atomic way, one isolated 
> >>feature at a  time. This is understandable and the right way to 
> >>start, but that vast
> >majority
> >> of bugs and issues Readium encounters are more complex 
> >> interactions, or odd variants of features.  I cannot offhand recall 
> >> a single bug we have encountered lately that was found through, or 
> >> could be tested with, the EPUB TestSuite.  To be clear, the TS is 
> >> very valuable, it simply doesnıt
> >go
> >> anywhere near far enough.
> >> 3. There are few if any failure cases.  Or stress cases. The TS is 
> >> so
> >constructed
> >> such that the tests are clean and well-behaved.  If only the 
> >>content we  received from all and sundry were equally well-behaved. 
> >>:-) The TS should  also be testing the RS to see how they handle 
> >>errors.  In some cases, the  EPUB spec does in fact specify how 
> >>errors or bad and/or illegal content  should be handled, but the TS 
> >>doesnıt test these. One small example:
> >>For
> >> reasons none of us who were there can recall, declarative SVG 
> >>animation is  not allowed in EPUB files.  EPUBCheck dutifully 
> >>complains about
> >(declarative)
> >> SVG animation, but the TS doesnıt test that such content is ignored.
> >>The
> >> overhead to disallow such content in a RS is quite large, so 
> >>Readium
> >simply
> >> doesnıt bother and SVG animation works fine even though it should 
> >>be rejected.  Again, to be clear, the TS is an awesome batch of 
> >>work, it just
> >isnıt
> >> enough.
> >>
> >> My two cents, FWIW.
> >>
> >> Ric
> >>
> >>
> >>
> >> On 12/27/17, 4:23 AM, "Daniel Glazman"
> >> <daniel.glazman@disruptive-innovations.com> wrote:
> >>
> >> >Le 26/12/2017 à 19:02, Bill McCoy a écrit :
> >> >
> >> >> That being said I think you're engaging in a logical fallacy by 
> >> >>claiming that a single mistake by the IDPF EPUB WG over a year 
> >> >>ago implies that we must now radically rethink the way W3C 
> >> >>Publishing@W3C groups are structured in general, as well as how 
> >> >>ongoing development of EPUB 3 will be handled specifically.
> >> >
> >> >I respectfully disagree 100%. If the market is already unable to 
> >> >digest
> >> >3.1 six years after the release of 3.0, when will it be able to 
> >> >digest 4.0, Web Publications, Portable Web Publications and more?
> >> >Given the EPUB 3 situation, any investment now on WP, PWP, EPUB4 
> >> >is too early and too expensive. The EPUB market is just unable to 
> >> >absorb a new spec every three years. Our whole model is flawed.
> >> >
> >> >I suggest we take that chartered time to make a 3.x version 
> >> >conform to regular W3C exit criteria, with a test suite and real 
> >> >implementation reports, and reach W3C REC (instead of "Recommended 
> >> >Specification" IDPF
> >> >designation) instead of diving into useless expensive blue-sky dreams.
> >> >
> >> >*THAT* would be *immensely* useful to *everyone*.
> >> >
> >> >Let me repeat myself on one important point: 
> >> >backwards-compatibility requirements of EPUB increase as the EPUB 
> >> >market increase. They will become mandatory soon. An ebook is an 
> >> >ebook is an ebook. Nobody wants to maintain several technical 
> >> >versions of the same ebook for a given technology, nobody wants to 
> >> >update (at a cost) ebooks published eons ago that should just 
> >> >work, nobody wants to leave legacy reading systems unattended 
> >> >because it could kill the ecosystem, and we're not even able to 
> >> >get rid of our ugliest design choices any more. Hence my
> >> >conclusion: EPUB is a dead end. A technological disruption will 
> >> >happen, it's not a question of "if" any more but only a question 
> >> >of
> "when".
> >> >
> >> ></Daniel>
> >> >
> >>
> >>
> >
> >
>

Received on Friday, 29 December 2017 19:18:22 UTC