Re: EARL structure and processing model

On Fri, 01 Apr 2005 22:38:32 +1000, Shadi Abou-Zahra <shadi@w3.org> wrote:

> Based on the discussions in several threads, it seems there are two  
> distinct but not separate topics: the structure of EARL, and the  
> processing model for tools. In the following, I will try to explain  
> these two in more detail.

Yep. This is a helpful discussion basis. Quite a lot of comments below...

> * EARL Structure
> There are several Classes and Attributes in EARL, some of them already  
> exist in the current draft, other may need to be defined. These are some  
> of the entities which have been discussed recently:
>
> - Subject
> Currently, the Subject is basically a URL of the page but it is  
> imaginable to also use XPath in order to allow more granularity. A URL  
> would be a fallback (that is compatible with XPath) in case the document  
> does not support XPath (tag soup or non-markup document). More about  
> when a tool should output what is described in the processing model.
>
> - Location
> Many tool developers already added such a construct into the EARL their  
> tools output. Indeed, such a construct is needed if the Subject remains  
> coarse as it currently is. However, it makes more sense to try and tie  
> the "location" aspect directly into the subject, for example by using  
> XPath.

I don't think so. I think we should identify the subject as an object, and  
where possible add further information (a new property) that describes  
where in the subject a particular assertion is focused. This makes it  
easier to take a collection of assertions about a single subject and know  
that they are describing the same thing, since otherwise we get into nasty  
hacks of trying to chop up URIs according to some heuristic or other (a  
bad idea).

RDF makes this important. For RDF, http://www.example.com/page#part1 and  
http://www.example.com/page#part1a may be completely unrelated - I can say

   <earl:WebContent r:resource=" http://www.example.com/page#part1">
     <earl:reprOf r:resource="http://www.microsoft.com"/>
   </earl:Webcontent>
   <earl:WebContent r:resource=" http://www.example.com/page#part1a">
     <earl:reprOf r:resource="http://www.ibm.com"/>
   </earl:Webcontent>

just as easily as saying that these are pointers directly into some page  
at http://www.example.com/page (this is why EARL has the webContent and  
reprOf constructs. If you want to describe a tool you give it a URI, but  
you can't necessarily get the tool at that URI...)

> - TestCase
> Currently, the TestCase is a URI to a test (the test itself doesn't have  
> to be published though, it is just a unique name to identify a test).  
> Often a TestCase may be composed of several sub-tests (for example the  
> TestCase "WCAG-AA conformance" requires numerous sub-tests). It is  
> important that EARL addresses this hierarchy in some form.

Yep. I think this is best done using the OWL constructs that you posted  
last year. For the price of a few lines of XML per subtest we don't have  
to rely on anything except standard OWL processing, rather than requiring  
the logic be built directly into each EARL processing tool. Conversely, if  
you are building the logic in already you have a free set of code for some  
standard OWL constructs that you can re-use in any OWL...

> - Evidence
> Has been proposed as a mechanism to describe the sub-tests carried out  
> for a TestCase. Basically it is an RDF list with all sub-tests executed.  
> It is also imaginable that this list is directly incorporated into the  
> TestCase construct. There may also be other solutions which we need to  
> develop in the new EARL version.

I don't think that we should expect the subtests to be directly  
incoroporated into a TestCase construct, although using the OWL construct  
mentioned above the designer of a test, or someone who creates a  
particular conformance profile they are looking for can identify a  
possible set of tests that satisfy the requirement.

The reason why I want evidence to include the rules it uses as well as the  
test results, is so you can compare evaluations of the same overall result  
which were done differently - say by different tools which each have their  
own mix of automation and manual guidance. You can also use two sets of  
evidence that you believe, but that were done according to different  
rules, to see if you have enough information to make a claim according to  
a third rule. For example, something that passes WCAG 1 double-A and  
Section 508 may or may not pass a particular level of WCAG 2. If you have  
pointers to the test results, you can go back and see if they are enough  
to make the WCAG 2 claim or if you need to do more testing.

> - Confidence
> The usage of "Confidence" is quite unclear and needs to be defined in  
> the processing model. For example, it could be tied to the  
> TestCase/Evidence structures, and how these tests were carried out (e.g.  
> automatic / manual / heuristic).

Yeah, I am not at all clear that confidence in a test result is really  
useful as part of EARL, because it is too hard to express it in a way that  
is interoperable.

> - Message
> This is purely for human consumption and should not include any  
> information that may need to be processed by machines. In fact,  
> sometimes it make sense to just point a URI to the error message within  
> a published test description. Also, messages need to support any natural  
> language.

Yeah, for the new semi-automatic version of Hera we have standard messages  
for automatic tests. Being RDF it is really easy to localise:

<earl:message r:resource="#message27"/>

<r:Description r:about="#message27">
   <s:label xml:lang="es">Nada funciona</s:label>
   <s:label xml:lang="en">Nothing works</s:label>
</r:Description>

this allows other people to publish their translations of the messages,  
like

<r:Description r:about="http:/www.sidar.org/hera/messages#message27">
   <s:label xml:lang="fr">Rien ne marche</s:label>
   <s:label xml:lang="en-AU">She's totally rooted mate</s:label>
</r:Description>

and we can collect that to localise a report without changing the tool  
itself.

> * EARL Processing Model
> Very closely related, we need to define a processing model for how to  
> use the structure. This would help developers and considerably reduce  
> fragmentation of the specification by incompatible interpretations. To  
> describe the "location" of an error, there are probably two key  
> components to consider, and tools will need to go into different "modes"  
> accordingly:

[Couple of good examples.]

> * Conclusion
> It seems that we need to do considerable work on the "Processing Model"  
> level which will throw down requirements onto the underlying  
> "Structure". For example, for "locating" an error we need to list the  
> different "situations", and what to do in such cases. This will focus  
> the scope of the Subject/Location structures, and how they should look  
> like in detail. Similar approach should be applicable to  
> TestCase/Evidence as well as Confidence discussions.

Yep.

cheers

Chaals

-- 
Charles McCathieNevile                      Fundacion Sidar
charles@sidar.org   +61 409 134 136    http://www.sidar.org

Received on Friday, 1 April 2005 13:28:02 UTC