EARL vs. our desired test results format

After reviewing our latest statement of desired test results format,
versus EARL and what I know of it, I think we won't be able to
reasonably express what we want in EARL. My reasons are these...

1. EARL does not have a notion of test suites as far as I can tell --
groups of tests. We have three levels here really -- the overall test
of a resource's mobileOK-ness, comprised of many mobileOK test result
outcomes, each comprised of several warns and failures. Right now we
sort of finesse this by pretending the overall test isn't a test, but
now that we have a third level this is problematic.

2. No suitable notion of "position" and code snippet. There's a notion
of a pointer to the source doc but it's an XPath  pointer as far as I
can tell.

3. EARL defines no real XML serialization. Not like we can't make up
the obvious one, but, it's a point.

4. EARL in XML is verbose and hard to understand. Here's an example result:

<earl:TestResult rdf:about="#result">
  <earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
  <dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>The <code>table</code> element is not allowed to appear
        inside a <code>p</code> element</p>
    </div>
  </dc:description>
  <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date">2006-08-13</dc:date>
  <earl:pointer rdf:resource="#xpointer"/>
  <earl:info rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>It seems the <code>p</code> element has not been closed</p>
    </div>
  </earl:info>
</earl:TestResult>

This is a more general knock of RDF in XML. Or maybe RDF. Ouch! But
really, this is the sort of output we want developers to use. They
need to make sense of it. We have to write a schema. Ick.

5. EARL has no notion of warnings, though we can conflate it with
"info" without too much trouble. Still a point.


So... I suggest we jump to a simpler, custom-built XML format like the
one we sketched last week. I don't think it's because EARL is
deficient, but because we have pretty detailed test output needs, and
also because it's RDF-ness just adds complexity here that doesn't help
-- well, that's the view from my brain, which has never really
"gotten" the point of RDF.


Sean

Received on Tuesday, 19 June 2007 01:43:51 UTC