RE: EARL vs. our desired test results format

As a matter of principle, I think it is desirable for us to stick with
W3C formats and extend where necessary. However, I also think there are
practical obstacles. The principal obstacle I see is that none of us on
this team are au fait with RDF. In the course of development of this
result format I imagine that there will be a fair amount of detailed
to-ing and fro-ing. The fact that we are collectively going to struggle
with even basic things like sub-classing is not really a good start.

In line with Roland's suggestion, I think it will be altogether more
practical for us to work in our own lingua franca and look at turning
this into EARL later, no doubt with a little help from our friends.

Jo

> -----Original Message-----
> From: public-mobileok-checker-request@w3.org [mailto:public-mobileok-
> checker-request@w3.org] On Behalf Of Shadi Abou-Zahra
> Sent: 19 June 2007 11:43
> To: Sean Owen
> Cc: public-mobileok-checker@w3.org; public-wai-ert@w3.org
> Subject: Re: EARL vs. our desired test results format
> 
> 
> Hi Sean, checker folks,
> 
> It's understandable if you can not wait for EARL and want to cook your
> own soup. However, I think there are some misunderstandings about EARL
> and I'd like to clarify some below. Also, I think it may be worth
while
> to consider extending EARL where it doesn't meet your specific goals
> rather than reinvent everything from scratch. Some of these additions
> may eventually make into EARL at some point too.
> 
> 
> Sean Owen wrote:
> > 1. EARL does not have a notion of test suites as far as I can tell
--
> > groups of tests. We have three levels here really -- the overall
test
> > of a resource's mobileOK-ness, comprised of many mobileOK test
result
> > outcomes, each comprised of several warns and failures. Right now we
> > sort of finesse this by pretending the overall test isn't a test,
but
> > now that we have a third level this is problematic.
> 
> This is the beauty of RDF -you can extend EARL with a custom test case
> description vocabulary. EARL provides a TestCriterion stub with two
> "built-in" sub-classes TestRequirement and TestCase. We think that
these
> terms, while not closely defined, can be further extended to serve
many
> different situations including mobileOK. It would be useful feedback
if
> you can let us know why these terms can not be extended to serve your
> specific needs.
> 
> 
> > 2. No suitable notion of "position" and code snippet. There's a
notion
> > of a pointer to the source doc but it's an XPath  pointer as far as
I
> > can tell.
> 
> Actually EARL provides many different types of pointers, including
code
> (and byte) snippets. Sadly this work is currently going a little
slower
> than expected but you can see first attempts in the "Pointer Methods
in
> RDF" document:
>   - <http://www.w3.org/WAI/ER/Pointers/WD-Pointers-20070222>
> 
> If you can not wait for this pointer vocabulary then you can just
create
> your own as sub-classes of earl:pointer. You can focus your energy on
> creating the new terms rather than reinventing the whole framework.
> 
> 
> > 3. EARL defines no real XML serialization. Not like we can't make up
> > the obvious one, but, it's a point.
> 
> EARL does not define an XML Schema or DTD, but we use RDF/XML as the
XML
> serialization. We are also currently working on a comment on this
draft
> to provide an XML Schema. There is no group decision on this yet but
if
> you develop your own XML Schema for EARL we may be inclined to adopt
it.
> 
> 
> > 4. EARL in XML is verbose and hard to understand. Here's an example
> result:
> > [...]
> > This is a more general knock of RDF in XML. Or maybe RDF. Ouch! But
> > really, this is the sort of output we want developers to use. They
> > need to make sense of it. We have to write a schema. Ick.
> 
> Granted. We tried to simplify EARL as much as possible but this is
> clearly the downside of RDF/XML. On the other hand, how many users
> really need to read the XML all the time?
> 
> 
> > 5. EARL has no notion of warnings, though we can conflate it with
> > "info" without too much trouble. Still a point.
> 
> We've had extensive debate about this in the past and decided that a
> warning is usually a subclass of pass. As I understood, also in
mobileOK
> a WARN is not a real result but an informative message before a PASS
or
> FAIL result. This is why we've added "info", to serve these purposes.
If
> you really need a warning *result* then just sub-class it as needed.
> 
> 
> > So... I suggest we jump to a simpler, custom-built XML format like
the
> > one we sketched last week. I don't think it's because EARL is
> > deficient, but because we have pretty detailed test output needs,
and
> > also because it's RDF-ness just adds complexity here that doesn't
help
> > -- well, that's the view from my brain, which has never really
> > "gotten" the point of RDF.
> 
> The primary motivation for creating EARL as an RDF format is to give
it
> the necessary flexibility and extensibility, yet provide a sufficient
> framework to exchange basic information on test results.
> 
> Anyway, as mentioned earlier this is not an attempt to convince you
but
> to clarify some of the issues. We would be very interested to learn
more
> about why EARL does not satisfy your specific needs and happier to
learn
> about any extensions you develop.
> 
> Regards,
>    Shadi
> 
> 
> --
> Shadi Abou-Zahra     Web Accessibility Specialist for Europe |
> Chair & Staff Contact for the Evaluation and Repair Tools WG |
> World Wide Web Consortium (W3C)           http://www.w3.org/ |
> Web Accessibility Initiative (WAI),   http://www.w3.org/WAI/ |
> WAI-TIES Project,                http://www.w3.org/WAI/TIES/ |
> Evaluation and Repair Tools WG,    http://www.w3.org/WAI/ER/ |
> 2004, Route des Lucioles - 06560,  Sophia-Antipolis - France |
> Voice: +33(0)4 92 38 50 64          Fax: +33(0)4 92 38 78 22 |

Received on Tuesday, 19 June 2007 15:28:31 UTC