Using EARL for Validator Web Service Format

Hi all.

[ This message is cross-posted to the Markup Validator, CSS  
Validator, Unicorn and qa-dev lists, with a reply-to set to the list  
with the largest number of participants. Apologies for the cross- 
post, I hope that was the best way to include all parties involved.  
Please respond in the www-validator list only. Thank you. ]

In the context of discussions around "a simpler web service response  
format" for our validators and for unicorn [0], we recently had a  
look at EARL [1], which has been evolving a lot recently, and will  
soon get in last call.

[0] Thread starting at:
http://lists.w3.org/Archives/Public/www-validator/2006Dec/0053.html
continued at:
http://lists.w3.org/Archives/Public/www-validator/2007Jan/0004.html

[1] http://www.w3.org/WAI/ER/

The validators are an obvious customer for EARL. Actually, EARL  
identifies from the introduction of the spec the example of a  
validator output using EARL (See [2], Example 2) as one of their  
basic use cases.

[2] http://www.w3.org/WAI/ER/EARL10/WD-EARL10- 
Schema-20070226#testresults

At this point, only one of the validators produced by W3C has an  
output in EARL [3], and it's a very old version of EARL. Updating  
that is on the radar, and I'm thinking it could very well replace the  
format we had tentatively designed for unicorn, if it allows a  
better, simpler (and standardized) format to be used.

[3] http://validator.w3.org/docs/users.html#Output


The basic model of EARL takes an assertor, checking a test case  
against a test subject and making an assertion on the test result. So  
far, so good.

A few of the questions we asked ourselves while starting to learn  
about EARL were:

Q1) how could we use earl and have a way to show, in the results, the  
list of messages (errors, warnings) that usually come with validation  
reports? Would we have to extend it with another namespace?

earl:info could do the job.
[[
     Additional information beyond the description of the result. For  
example warnings or other informative messages that may help a reader  
better understand the result. It is recommended to use Literal values  
for such additional messages.
]]
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#testresult

I think the recommendation to parse as Literal is new. Which would  
make more sense, if we want to list errors, warnings and info  
messages, possibly along with a count info? Use divs, ol, li in a  
Literal? Or use something else, in another namespace? What would be  
simpler, what would be more flexible?


Q2) In the case of the CSS validator, when checking an html page for  
example, the results are given for that page and the CSS files it  
links to. What, then, is the "test subject"?

Still unsure. I think it would be best to keep the page given by the  
user to be checked as the test subject, but it would also be useful  
to identify the fact that there are sub-subjects.

Maybe the CSS validator could make assertions on the single CSS  
documents, and have information on the fact that they are "part" of a  
wider test subject with dct:hasPart or dct:isPartOf:
[[
dct:hasPart
     Relationship to other subjects that are part of this subject
dct:isPartOf
     Relationship to other subjects of which this subject is a part of
]]
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#testsubject


Q3) What are our Test Criteria?

I suppose we'd have to have test criterion URIs for each of the  
validations.
And consider the "tasks" of Unicorn (general conformance, XHTML+CSS 
+...) as earl:TestRequirement
[[
     A higher-level requirement that is tested by executing one or  
more sub-tests. For example WCAG 1.0 Checkpoint 1.1 which is tested  
by executing several sub-tests and combining the results.
]]
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#testable


Q4) What does Unicorn do? Collect and present assertions, or is it an  
assertor too?

Would it be interesting to consider it as a compound assertor?
http://www.w3.org/WAI/ER/EARL10/WD-EARL10- 
Schema-20070226#compoundassertor
but that's only interesting if Unicorn itself has an earl OUTput,  
while for now I think the interesting part is to have the various  
observers have EARL outputs that are used by unicorn as INput to be  
merged and processed.

Q5) Does the Outcome hard-coded pre-defined values cover all our needs?

Would be nice asking the ER WG to add some, if not.
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Schema-20070226#outcome
At least, EARL would be more precise than what the Unicorn format  
currently has, because the Unicorn format says pass/fail without  
describing what it passes or fails. It is unclear whether it means  
"did not pass validation" or "could not validate". EARL has values  
for both.

Q6) Misc Notes

* earl:sourceCopy could be interesting for "direct input", although  
I'm unsure whether it would be useful to copy massive chunks of text  
around like that. Could we just use a hash as earl:subject? We need  
to find ways to identify the subject for direct input and file  
upload, in any case.

* Our assertors are earl:Software acting in earl:automatic mode.

* We can probably make the output formats short by using URIs  
refering to full description for the assertors and tests, instead of  
having the full description and classes each time.

* ... I had more notes, but I need to find the deadwood I had written  
them on.


Hope this gives us an interesting material for discussion, for now.

olivier
-- 
olivier Thereaux - W3C - http://www.w3.org/People/olivier/
W3C Open Source Software: http://www.w3.org/Status

Received on Friday, 2 March 2007 06:07:00 UTC