EARL 1.0 Specification
ERWG Draft Note 15 December 2001
- Latest Version:
- http://www.w3.org/xxxx/xx/xxxxxxxx
- Editor:
- Sean B. Palmer (sean@mysterylights.com)
This document is a general specification for the use of EARL 1.0. This is not a wholly traditional specification, in that it constitutes both a primer and a specification all-in-one.
This document is currently provided only as a note by the author for review by the W3C WAI ER Working Group, and by any interested parties. It is neither endorsed by the Working Group, nor constitutes any formal representation of the views and/or consensus of the group.
Feedback is most welcome, and should be directed to the ER mailing list, w3c-wai-er-ig@w3.org (archives). Please send minor editorial corrections and comments directly to the editor.
All names and URIs used herein are purely examples, and their use does not constitute any kind of endorsement or recommendation etc. of those resources.
The Evaluation And Report Language (EARL) is a generic machine-friendly model for storing evaluations of Web content, tools, and other "things". The model itself ranges from a high level architecture involving context information, and the nature/test subject/criteria of the assertion, to low level granular information including test case metadata, mechanisms to point to a range of test subjects, and a multitude of documentation/utility data.
EARL is therefore a set of structures and terms grounded in the Web, that can then be expressed in a number of ways. The ER Working Group chose RDF as the default method of recording EARL evaluations, and as such created a normative RDF Schema for the "language".
EARL is an intensely practical language, and although technical in some areas, one of the main objectives of EARL is that it can be used as an interoperable method of storing evaluation data. Therefore it is an important requirement that it be simple enough to implement with only a light understanding of the full model.
Because of these objectives (and the collorary requirements), EARL is a very powerful language; it does not limit one to making evaluations about Web pages only, for example. At the basic level, it enables any person, entity, or organization to evaluate anything (any "thing") against any criterion or set of criteria. At the deeper level, it provides "tools" for enabling people to be very specific about the nature of their evaluations. It is also highly extensible, and extensions are encouraged, mandated, and allowed for in the model as much as possible.
The EARL "model" is the syntax independent, abstract framework for making EARL evaluations.
The model must be simple enough to understand, rigid enough so that minor changes do not affect the gist of the data, and extensible. Contrary to the Perl motto, deciding on how to approach something in a singular manner is advantageous - coming to consensus on a standard method for (e.g) identifying a set of pages is very useful, for it builds interoperability. It consensus based standardization rather than proprietary centralization.
To any evaluation, there are four major parts (a quad):-
In the EARL model, we call the latter three (the test subject, the validity, and the test case) the assertion, and the whole set (the assertion plus the context information) the evaluation.
It is very easy to provide prose examples that follow the above structures. Here are some very simple examples:-
A number of things become immediately evident. Firsly, there are a wide range of evaluation types, but they all involve similar sub-structures. Secondly, these are realistic scenarios, that have a range of purposes and uses. Thirdly, while the specific data types (for example, the date format for "17th December 2000" vs. "2001-05-17") differ in the example above, it would not be difficult to form consensus on a common set of datatypes to use, and nor would it be difficult to identify the range of datatypes (names, dates, URIs, etc.) used in general evaluations. This is another one of the objectives of EARL.
The EARL model is built around these two things: the structure of the evaluations, and the vocabulary and datatypes that one uses within this structure. Because we decided to express EARL as RDF, we found very easy ways of implementing both of these.
RDF (the Resource Description Framework) is the W3C's standard for metadata. Metadata are data about data. To learn EARL, you do not need to have a strong knowledge of how RDF works, and all of its potential applications, but thankfully, the basic model as relates to EARL is very simple to learn.
In RDF, a document consists of a set of statements, and all statements are triples; that is, made up of three pieces of information. This is almost directly analogous to the sentence "noun verb object" in natural language. Examples of triples are "John likes Chocolate", "MyPage hasTitle 'My Page'", and "SomeEvent date 2001-07-05". There are no exceptions to using triples in RDF, and hence the model is extremely stable and well defined.
Because there are actually four terms in an EARL evaluation (context, test subject, validity, and test case) we have to add a term, and then employ a method called "reification" that lets us use one triple as the subject of another; in other words, to nest that triple within a parent triple. To demonstrate, here is the first example presented again as prose:-
Mary Thompson asserts that on 17th December 2000 (context), the page at http://example.org/ (test subject) passed (validity) checkpoint 1.1 in the Web Content Accessibility Guidelines (test case)
The "Assertion" (consisting of the test subject, validity, and test case) can be modelled quite straightforwardly as a triple:-
http://example.org/ passed WCAG1.1
To add the context information, we simply use the verb "asserts" as a property in the evaluation. (In fact the term in the middle of a triple in RDF is termed the "predicate". All predicates are necessarily properties, but properties are not restricted to only being used in the predicate position of a triple: they can be subjects (the first term) or objects (the third term) as well). For example:-
Mary_Thompson asserts (http://exmaple.org/ passed WCAG1.1) W3C_Validator asserts (http://example.org/ fails XHTML_compliance)
We have used the assertion as the object in our evaluation.
In summary then, we model the four parts of the EARL evaluation by adding one term ("asserts"), and then using the assertion triple as an object.
Because there is no limit to the amount of triples that we can use, we can model all of the pieces of the evaluation using just triples. As an example, here is the last example:-
Bobby Smith (context) asserts that his browser, CoolBrowser v 1.0 revision date 2001-05-17 (test subject) passed (validity) the CoolBrowser test suite for ensuring CSS 1.0 support
as triples:-
Bobby_Smith asserts (CoolBrowser passed CSSTest) Bobby_Smith name "Bobby Smith" CoolBrowser version "1.0" CoolBrowser revisionDate "2001-05-17" CoolBrowser a Browser CSSTest a TestSuite CSSTest ensures "CSS 1.0 support"
The final important part of the RDF model (and hence, the EARL model) is how to avoid name conflicts. Anyone can just come along and create names like "Bobby_Smith" and use them, and therefore when there are two Bobby Smiths, there will be a name conflict. The solution to this problem is to use URIs (Uniform Resource Identifiers) as the terms in the triples. So, instead of:-
Bobby_Smith name "Bobby Smith"
we may use something like:-
http://www.bobbysmith.org/Bobby_Smith http://schema.org/name "Bobby Smith"
We still allow literal string values (enclosed in quotes above) for names, dates, and other values. [N.B. datatypes are currently being discussed by the RDF Core Working Group.]
To "serialize" is synonymous with the act of coming up with a standard format for expressing the EARL triples. RDF offers a few competing formats: XML RDF, NTriples, and Notation3. The ER WG decided that XML RDF (pending updates by the RDF Core working group) is to be the standard serialization for EARL. [TODO: rant about the benefits of Notation3.]
XML RDF was chosen because it is possible to access the model through a DOM, as well as being easy to transform with XSLT. The disadvantage is that it is difficult to author quickly, although it is still possible to use Notation3, and then convert it back into XML RDF at a later date.
All EARL terms begin with
"http://www.w3.org/2001/03/earl/1.0-test#".
This is a simple example of what EARL is like when drafted in XML RDF. Hopefully, it will be fairly simple to follow; the only new things presented being XML namespaces, etc. [TODO: more links to explanations!]
<rdf:RDF xmlns="http://www.w3.org/2001/03/earl/1.0-test#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" >
<Assertor>
<email rdf:resource="mailto:bob@example.org"/>
<name>Bob B. Bobbington</name>
<asserts rdf:parseType="Resource">
<rdf:subject>
<WebContent>
<reprOf rdf:resource="http://example.org/net/bob/"/>
<date>2001-05-07</date>
</WebContent>
</rdf:subject>
<rdf:predicate rdf:resource="http://www.w3.org/2001/03/earl/1.0-test#passes"/>
<rdf:object rdf:parseType="Resource">
<testCriteria rdf:parseType="Resource">
<rdfs:label>WCAG Single A Compliance Test</rdfs:label>
<suite rdf:resource="http://www.w3.org/TR/WCAG10/"/>
<level rdf:resource="http://www.w3.org/TR/WCAG10/#wc-priority-1"/>
<note>All Priority 1 checkpoints => WCAG Single-A compliance</note>
</testCriteria>
</rdf:object>
<testMode rdf:resource="http://www.w3.org/2001/03/earl/1.0-test#Manual"/>
</asserts>
</Assertor>
</rdf:RDF>
This roughly conforms to the following prose definition:-
The assertor named "Bob B. Bobbington" with email address bob@example.org asserts that the WebContent at http://example.org/net/bob/ on date 2001-05-07 passes the WCAG Single A Compliance test; i.e. that all priority 1 checkpoints have been covered. This was a manual evaluation.
Although many cases will warrant much more complicated examples, it is still possible to get into EARL right now by using the example above as a template. All variables are delimited in <var> tags (usually renders as italic).
It may be difficult to vizualize the triples that come from this instance of XML RDF EARL. To help, try pasting the example into the W3C's RDF Validator. The validator output both a list of the triples, and a diagram (in a choice of formats) of the data model. For example, here is the list of triples that the example above generates:-
@prefix earl: <http://www.w3.org/2001/03/earl/1.0-test#> . _:j24578 earl:email <mailto:bob@example.org> . _:j24578 earl:name "Bob B. Bobbington" . _:j24581 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> earl:WebContent . _:j24581 earl:reprOf <http://example.org/net/bob/> . _:j24581 earl:date "2001-05-07" . _:j24580 <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> _:j24581 . _:j24580 <http://www.w3.org/1999/02/22-rdf-syntax-ns#predicate> earl:passes . _:j24586 earl:label "WCAG Single A Compliance Test" . _:j24586 earl:suite <http://www.w3.org/TR/WCAG10/> . _:j24586 earl:level <http://www.w3.org/TR/WCAG10/#wc-priority-1> . _:j24586 earl:note "All Priority 1 checkpoints => WCAG Single-A compliance" . _:j24585 earl:testCriteria _:j24586 . _:j24580 <http://www.w3.org/1999/02/22-rdf-syntax-ns#object> _:j24585 . _:j24580 earl:testMode earl:Manual . _:j24578 earl:asserts _:j24580 . _:j24578 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> earl:Assertor .
Note that this "list" of triples is actually in another serialization of RDF, called Notation3.
It is very important to be clear about what is identified in EARL (cf. Identity). Because EARL allows anyone to evaluate anything against anything else, we set out special properties and classes to enable people to do this. You may have noticed the following snippet from the example EARL above:-
<rdf:subject rdf:parseType="Resource">
<rdf:type
rdf:resource="http://www.w3.org/2001/03/earl/1.0-test#WebContent"/>
<reprOf rdf:resource="http://example.org/net/bob/"/>
<date>2001-05-07</date>
</rdf:subject>
We call that the "Test Subject". In this case, we are creating what is
called an anonymous test subject - it has no URI. The above can be
read as "the 'thing' that is of type 'WebContent', is a reprOf
http://example[...]/bob/, and has date
'2001-05-07'". We will note how to give a proper URI-view to the
"thing" a little further down.
Let's describe more carefully what is going on in the example above. The
resource identified by the URI http://example.org/net/bob/ is
most likely going to be Bob's homepage. Bob's homepage is of course subject
to change, just like most pages on the Web it gets updated, fiddled about
with, and overhauled. Not only that, but people may be able to to content
and/or language negotiation.
In order to be clear about what is being identified, we have to add the extra properties. Instead of saying "we're evaluating Bob's homepage", we may say that we're evaluating a particular representation of Bob's homepage sent back on a certain date and time, with the format "text/html", the SHA hash iprjg058w5089w0pegojw, and the language "en". We obviously cannot force people to be accurate in their disambiguation of the test subject, and we leave it to the user to figure out just how much information they need to add.
Because one of the greatest powers of EARL is that it does not force you evaluate a certain type of thing, we are not constrained to evaluating representations of resources got over HTTP. For example, we can evaluate resources themselves, or perhaps something that isn't traditionally considered to be a generic document, like a User Agent, or accessiblity tool.
Say that we wanted to evaluate http://www.w3.org/ as a
resource by itself, or use a proper identifier for the TestSubject... the
following is the syntax that lets you do that:-
<rdf:RDF xmlns="http://www.w3.org/2001/03/earl/1.0-test#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<Assertor>
[...]
<asserts rdf:parseType="Resource">
<rdf:subject rdf:resource="http://example.org/#Blargh" />
[...]
</asserts>
</Assertor>
<rdf:Description rdf:about="http://example.org/#Blargh">
<rdf:type rdf:resource="http://www.w3.org/2001/03/earl/1.0-test#WebContent"/>
</rdf:Description>
</rdf:RDF>
Note how we put the additional information outside of the reified statement, and indeed outside of the entire evaluation. That's just the way XML RDF works. (Well, you could use rdf:about in there too).
Now that we have this special syntax, it is possible to talk about basically anything.
TODO: XPointers.
[ earl:reprOf <http://example.org/>;
earl:date "2001-10-17"; earl:mime "text/xml";
earl:xpointer "xpointer(id(\"toc\")/li[3])" ] .
Things with a Consistent Fragment Syntax would be O.K., too.
Either yyyy-mm-dd or yyyy-mm-dd hh:mm:ss. TODO:
check.
TODO: We could define something like a RELAX NG grammar here, but it's almost certainly not worth it. It would be nice to come up with some sort of half-decent schema validator, though, and provide it as a Web service.
Automatically generated using getdoc.n3.
The EARL vocabularly, and to some extent the EARL model, are extensible; that is, they allow to to add new terms or otherwise modify them to fit your own specific application demands more closely. Indeed, the level of specificity in EARL is often not sufficent enough to avoid extending EARL. Thus, EARL was built to be extended. You may think of it as a core set of structures and terms.
We generally expect two categoris of implementations for EARL:-
The latter method is very difficult to extend; all it can do is to ignore property values that it does not understand. The former is much more powerful, but must allow for the latter. Therefore, whilst EARL actively encourages extensions, these should be made keeping in mind the two different POVs above.
TODO: this section really needs thinking through and rewording. cf. Evolution.
For example, you may want to create your own ResultProperty, which has a custom level of confidence set. Let's say that you want to make an assertion that your page passes a certain checkpoint with a "medium" degree of confidence. To do that, you would have to create a new instance of ResultProperty, with a validity arc pointing to earl:Pass, and an confidence arc pointing to the new term "Medium". viz.:-
:myNewProperty rdf:type earl:ResultProperty; earl:validity earl:Pass; earl:confidence :Medium .
TODO: the set of result properties is actually explicitly enumerated, so we would have to change it to allow this.
Note that when you create a new property, you have to give it a name (this could be automatically generated), and put it in your own namespace. Do not under any circumstances invent new properties in the EARL namespace that don't exist.
You can then use the new property as normal:-
:Someone earl:asserts _:x . _:x rdf:subject :SomePage . _:x rdf:predicate :myNewProperty . _:x rdf:object :SomeCriteria .
TODO: How would EARL agents handle this? If they were fully RDF/SW-ized, then they'd be able to handle it just fine... but they won't be. We could perhaps offer CWM with EARL filters as a Web service, but that would introduce a certain amount of centralization. Still, deduct and then spit out the EARL 1.0 sounds quite useful.
In other words, conventionally, if you come across the property "x", and you don't understand it, then you don't understand it. With rules on the SW, you can say "well, this is an EARL result property, with a validity of pass, so it's roughly analogous to pass" - without having to worry about the exact semantics. It's partial understanding, but it's very difficult for anyone who is just trying to make EARL work on the basic level. The problem with powerful languages is that they need a certain level of power in the tools that grok them, and although that level is not all that great, it's still great enough to deter some.
One way of doing it would be to have inference to an extent - always search for the validity property if a predicate is used that you do not understand, but discard anything else. At least that way, you can operate and ignore the extensions.
It is important that the specification (at W3C) of a language ground the meaning of a document written in that language in terms of a technical specification.
EARL is an abstract set of terms, and an ontology, using the RDF model. An EARL document will always be an RDF document that uses predicates from the EARL vocabulary set, or classes as objects of an rdf:type relation in a triple. Strictly, for an RDF document to be considered an "EARL document", there must be one or more instances of earl:Evaluation asserted in the document.
Issues of MIME type, model theory, and like-issues are delegated to RDF Core. EARL has no special exceptions to any of the RDF principia.
Available as 1.0.n3.
The working members of the ER WG, and important contributors to EARL at some point during its development (whether they knew it or not) include: Harvey Bingham, Dan Brickley, Wendy A. Chisholm, Dan Connolly, Daniel Dardailler, Al Gilman, Len Kasday, Nick Kew, Jim Ley, William "Geeze" Loughborough, Charles "Chaals" McCathieNevile, Libby Miller, Dave Pawson, Chris Ridpath, and Aaron Swartz.
Space does not permit listing of the folk whom contributed to RDF, XML, the W3C infrastructure, and all of the other things that we have depended upon, but we thank them anyway.
Sean B. Palmer, $Id$