W3C home > Mailing lists > Public > public-html-testsuite@w3.org > February 2013

Re: classify every section in the spec

From: James Graham <jgraham@opera.com>
Date: Thu, 28 Feb 2013 10:15:33 +0100
Message-ID: <512F2035.9040709@opera.com>
To: public-html-testsuite@w3.org
On 02/28/2013 03:12 AM, Michael Dyck wrote:

> The classes above are synthesized from 3 more basic conditions:
>   (1) whether the section has conformance requirements;
>   (2) whether there are tests for that section; and
>   (3) whether there are known interop issues pertaining to that section.
> Note also that these conditions can change when (respectively):
>   (a) the spec is edited;
>   (b) tests are submitted or edited; or
>   (c) a new version of a browser is released.
> Robin's coverage report already tells us (1) and (2), and (I gather) can
> be regenerated at will to reflect changes due to (a) and (b).
> Thus, rather than classifying sections into A/B/C/D, you could get the
> same information with less work (both upfront and ongoing) by just
> classifying them wrt (3), presence/level of known interop issues.
> (I'm assuming this has to be done by humans, i.e., you can't easily write
> a script to deduce the level of known interop issues for a section. Can
> you? I wondered whether we could import data from caniuse.com, but Kris
> said no.)

In an ideal world, of course, we would find out about known 
interoperability issues from the tests themselves. Of course, in many 
cases we might have out-of-band data indicating interop. problems but no 
tests that show them. This is, of course, a giant red flag that whatever 
tests we have are providing insufficient coverage of that part of the 
spec., irrespective of what other metrics say. So I am certainly in 
favour of utilizing such data where it is available.

The question is where to source the data. I have the impression that 
often the author community "feels" like some part of a spec isn't 
reliable in a set of browsers, but other than caniuse.com (which seems 
fine to use to me, although the data isn't really granular enough), this 
isn't recorded anywhere. And there isn't a culture of turning "I feel 
like Appcache is an interop disaster" into "here are a bunch of tests 
for Appcache to show the problems I've been having".
Received on Thursday, 28 February 2013 09:16:03 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 6 January 2015 20:14:32 UTC