Re: EARL vs. our desired test results format

Hi Sean, checker folks,

It's understandable if you can not wait for EARL and want to cook your 
own soup. However, I think there are some misunderstandings about EARL 
and I'd like to clarify some below. Also, I think it may be worth while 
to consider extending EARL where it doesn't meet your specific goals 
rather than reinvent everything from scratch. Some of these additions 
may eventually make into EARL at some point too.


Sean Owen wrote:
> 1. EARL does not have a notion of test suites as far as I can tell --
> groups of tests. We have three levels here really -- the overall test
> of a resource's mobileOK-ness, comprised of many mobileOK test result
> outcomes, each comprised of several warns and failures. Right now we
> sort of finesse this by pretending the overall test isn't a test, but
> now that we have a third level this is problematic.

This is the beauty of RDF -you can extend EARL with a custom test case 
description vocabulary. EARL provides a TestCriterion stub with two 
"built-in" sub-classes TestRequirement and TestCase. We think that these 
terms, while not closely defined, can be further extended to serve many 
different situations including mobileOK. It would be useful feedback if 
you can let us know why these terms can not be extended to serve your 
specific needs.


> 2. No suitable notion of "position" and code snippet. There's a notion
> of a pointer to the source doc but it's an XPath  pointer as far as I
> can tell.

Actually EARL provides many different types of pointers, including code 
(and byte) snippets. Sadly this work is currently going a little slower 
than expected but you can see first attempts in the "Pointer Methods in 
RDF" document:
  - <http://www.w3.org/WAI/ER/Pointers/WD-Pointers-20070222>

If you can not wait for this pointer vocabulary then you can just create 
your own as sub-classes of earl:pointer. You can focus your energy on 
creating the new terms rather than reinventing the whole framework.


> 3. EARL defines no real XML serialization. Not like we can't make up
> the obvious one, but, it's a point.

EARL does not define an XML Schema or DTD, but we use RDF/XML as the XML 
serialization. We are also currently working on a comment on this draft 
to provide an XML Schema. There is no group decision on this yet but if 
you develop your own XML Schema for EARL we may be inclined to adopt it.


> 4. EARL in XML is verbose and hard to understand. Here's an example result:
> [...] 
> This is a more general knock of RDF in XML. Or maybe RDF. Ouch! But
> really, this is the sort of output we want developers to use. They
> need to make sense of it. We have to write a schema. Ick.

Granted. We tried to simplify EARL as much as possible but this is 
clearly the downside of RDF/XML. On the other hand, how many users 
really need to read the XML all the time?


> 5. EARL has no notion of warnings, though we can conflate it with
> "info" without too much trouble. Still a point.

We've had extensive debate about this in the past and decided that a 
warning is usually a subclass of pass. As I understood, also in mobileOK 
a WARN is not a real result but an informative message before a PASS or 
FAIL result. This is why we've added "info", to serve these purposes. If 
you really need a warning *result* then just sub-class it as needed.


> So... I suggest we jump to a simpler, custom-built XML format like the
> one we sketched last week. I don't think it's because EARL is
> deficient, but because we have pretty detailed test output needs, and
> also because it's RDF-ness just adds complexity here that doesn't help
> -- well, that's the view from my brain, which has never really
> "gotten" the point of RDF.

The primary motivation for creating EARL as an RDF format is to give it 
the necessary flexibility and extensibility, yet provide a sufficient 
framework to exchange basic information on test results.

Anyway, as mentioned earlier this is not an attempt to convince you but 
to clarify some of the issues. We would be very interested to learn more 
about why EARL does not satisfy your specific needs and happier to learn 
about any extensions you develop.

Regards,
   Shadi


-- 
Shadi Abou-Zahra     Web Accessibility Specialist for Europe |
Chair & Staff Contact for the Evaluation and Repair Tools WG |
World Wide Web Consortium (W3C)           http://www.w3.org/ |
Web Accessibility Initiative (WAI),   http://www.w3.org/WAI/ |
WAI-TIES Project,                http://www.w3.org/WAI/TIES/ |
Evaluation and Repair Tools WG,    http://www.w3.org/WAI/ER/ |
2004, Route des Lucioles - 06560,  Sophia-Antipolis - France |
Voice: +33(0)4 92 38 50 64          Fax: +33(0)4 92 38 78 22 |

Received on Tuesday, 19 June 2007 10:43:31 UTC