This document is a general specification for the use of EARL 1.0. This is not a wholly traditional specification, in that it constitutes both a primer and a specification all-in-one.
This document is currently provided only as a note by the author for review by the W3C WAI ER Working Group, and by any interested parties. It is neither endorsed by the Working Group, nor constitutes any formal representation of the views and/or consensus of the group.
Feedback is most welcome, and should be directed to the ER mailing list, w3c-wai-er-ig@w3.org (archives). Please send minor editorial corrections and comments directly to the editor.
All names and URIs used herein are purely examples, and their use does not constitute any kind of endorsement or recommendation etc. of those resources.
The Evaluation And Report Language (EARL) is a generic model for storing evaluations of resources. The model itself ranges from a high level architecture involving context information, and the nature/test subject/criteria of the assertion, to low level granular information including test case metadata, mechanisms to point to a range of test subjects, and so on.
EARL is therefore a set of structures and datatypes that can then be expressed in a number of ways. The ER Working Group chose RDF as the default method of recording EARL evaluations, and as such created a normative RDF Schema for the "language".
EARL is an intensely practical language, and although technical in some areas, one of the objectives of EARL is that it can be used as an interoperable method of storing evaluation data, and therefore it is a requirement that it be simple enough to implement with only a light understanding of the full model.
Because of these objectives (and the collorary requirements), EARL is a very powerful language; it does not limit one to making evaluations about Web pages only, for example. At the basic level, it enables any person, entity, or organization to evaluate any resource (any "thing") against any criterion or criteria. At the deeper level, it provides "tools" for enabling people to be very specific about the nature of their evaluations. It is also highly extensible, and extensions are encouraged, mandated, and allowed for in the model as much as possible.
The EARL "model" is the syntax independent, abstract framework for making EARL evaluations.
[Rough section]
To any evaluation, there are four major parts (a quad):-
In the EARL model, we call the latter three (the test subject, the validity, and the test case) the assertion, and the whole set (the assertion plus the context information) the evaluation.
It is very easy to provide prose examples that follow the above structures. Here are some very simple examples:-
A number of things become immediately evident. Firsly, there are a wide range of evaluation types, but they all involve similar sub-structures. Secondly, these are realistic scenarios, that have a range of purposes and uses. Thirdly, while the specific data types (for example, the date format for "17th December 2000" vs. "2001-05-17") differ in the example above, it would not be difficult to form consensus on a common set of datatypes to use, and nor would it be difficult to identify the range of datatypes (names, dates, URIs, etc.) used in general evaluations. This is another one of the objectives of EARL.
The EARL model is built around these two things: the structure of the evaluations, and the vocabulary and datatypes that one uses within this structure. Because we decided to express EARL as RDF, we found very easy ways of implementing both of these.
RDF (the Resource Description Framework) is the W3C's standard for metadata. Metadata is data about data. To learn EARL, you do not need to have a strong knowledge of how RDF works, and all of its potential applications, but thankfully, the basic model as relates to EARL is very simple to learn.
In RDF, all facts are triples; that is, three pieces of information. This is almost directly analogous to the sentence "noun verb object" in natural language. Examples of triples are "John likes Chocolate", "MyPage hasTitle 'My Page'", and "SomeEvent date 2001-07-05". There are no exceptions to using triples in RDF, and hence the model is extremely stable and well defined.
Because there are actually four terms in an EARL evaluation (context, test subject, validity, and test case) we have to add a term, and then employ a method called "reification" that lets us use one triple as the subject of another; in other words, to nest that triple within a parent triple. To demonstrate, here is the first example presented again as prose:-
Mary Thompson asserts that on 17th December 2000 (context), the page at http://example.org/ (test subject) passed (validity) checkpoint 1.1 in the Web Content Accessibility Guidelines (test case)
The assertion (consisting of the test subject, validity, and test case) can be modelled quite straightforwardly as a triple:-
http://example.org/ passed WCAG1.1
To add the context information, we simply use the verb "asserts" as a property in the evaluation. (In fact the term in the middle of a triple in RDF is termed the "predicate". All predicates are necessarily properties, but properties are not restricted to only being used in the predicate position of a triple: they can be subjects (the first term) or objects (the third term) as well). For example:-
Mary_Thompson asserts (http://exmaple.org/ passed WCAG1.1) W3C_validator asserts (http://example.org/ fails XHTML_compliance)
We have used the assertion as the object in our evaluation.
In summary then, we model the four parts of the EARL evaluation by adding one term ("asserts"), and then using the assertion triple as an object.
Because there is no limit to the amount of triples that we can use, we can model all of the pieces of the evaluation using just triples. As an example, here is the last example:-
Bobby Smith (context) asserts that his browser, CoolBrowser v 1.0 revision date 2001-05-17 (test subject) passed (validity) the CoolBrowser test suite for ensuring CSS 1.0 support
as triples:-
Bobby_Smith asserts (CoolBrowser passed CSSTest) Bobby_Smith name "Bobby Smith" CoolBrowser version "1.0" CoolBrowser revisionDate "2001-05-17" CoolBrowser a Browser CSSTest a TestSuite CSSTest ensures "CSS 1.0 support"
The final important part of the RDF model (and hence, the EARL model) is how to avoid name conflicts. Anyone can just come along and create names like "Bobby_Smith" and use them, and therefore when there are two Bobby Smiths, there will be a name conflict. The solution to this problem is to use URIs (Uniform Resource Identifiers) as the terms in the triples. So, instead of:-
Bobby_Smith name "Bobby Smith"
we may use something like:-
http://www.bobbysmith.org/Bobby_Smith http://schema.org/name "Bobby Smith"
We still allow literal string values (enclosed in quotes above) for names, dates, and other values. Datatypes are currently being discussed by the RDF Core Working Group.
Serialization refers to the act of coming up with a standard format for expressing the EARL triples. RDF offers a few competing formats: XML RDF, NTriples, and Notation3. EARL is serialization independent; i.e. it can use any "conformant" serializaton of RDF.
However, because we need to provide EARL examples, we must choose a particular serialization, and stick to it. In the author's current opinion, Notation3 is the easiest serialization to learn and use... however, the canonical form of EARL will be in XML RDF, and XML RDF is also the only serialization recommended by the W3C. Therefore, throughout this draft, we shall use XML RDF.
All EARL terms begin with "http://www.w3.org/2001/03/earl/1.00#".
[To do]
<rdf:RDF xmlns="http://www.w3.org/2001/03/earl/1.00#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<Assertor>
<email rdf:resource="mailto:bob@example.org"/>
<name>Bob B. Bobbington</name>
<asserts rdf:parseType="Resource">
<rdf:subject rdf:parseType="Resource">
<rdf:type
rdf:resource="http://www.w3.org/2001/03/earl/1.00#WebContent"/>
<testSubject rdf:resource="http://example.org/net/bob/"/>
<date>2001-05-07</date>
</rdf:subject>
<rdf:predicate rdf:resource="http://www.w3.org/2001/03/earl/1.00#passes"/>
<rdf:object rdf:parseType="Resource">
<testCriteria rdf:parseType="Resource">
<label>WCAG Single A Compliance Test</label>
<suite rdf:resource="http://www.w3.org/TR/WCAG10/"/>
<level rdf:resource="http://www.w3.org/TR/WCAG10/#wc-priority-1"/>
<note>All Priority 1 checkpoints => WCAG Single-A compliance</note>
</testCriteria>
</rdf:object>
<testMode rdf:resource="http://www.w3.org/2001/03/earl/1.00#Manual"/>
</asserts>
</Assertor>
</rdf:RDF>
This roughly conforms to the following prose defintion:-
The assertor named "Bob B. Bobbington" with email address bob@example.org asserts that the WebContent at http://example.org/net/bob/ on date 2001-05-07 passes the WCAG Single A Compliance test; i.e. that all priority 1 checkpoints have been covered. This was a manual evaluation.
It may be difficult to vizualize the triples that come from this instance of XML RDF EARL. To help, try pasting the example into the W3C's RDF Validator. The validator output both a list of the triples, and a diagram (in a choice of formats) of the data model. For example, here is the list of triples that the example above generates:-
@prefix earl: <http://www.w3.org/2001/03/earl/1.00#> . _:j24578 earl:email <mailto:bob@example.org> . _:j24578 earl:name "Bob B. Bobbington" . _:j24581 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> earl:WebContent . _:j24581 earl:testSubject <http://example.org/net/bob/> . _:j24581 earl:date "2001-05-07" . _:j24580 <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> _:j24581 . _:j24580 <http://www.w3.org/1999/02/22-rdf-syntax-ns#predicate> earl:passes . _:j24586 earl:label "WCAG Single A Compliance Test" . _:j24586 earl:suite <http://www.w3.org/TR/WCAG10/> . _:j24586 earl:level <http://www.w3.org/TR/WCAG10/#wc-priority-1> . _:j24586 earl:note "All Priority 1 checkpoints => WCAG Single-A compliance" . _:j24585 earl:testCriteria _:j24586 . _:j24580 <http://www.w3.org/1999/02/22-rdf-syntax-ns#object> _:j24585 . _:j24580 earl:testMode earl:Manual . _:j24578 earl:asserts _:j24580 . _:j24578 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> earl:Assertor .
This list of triples is actually in another serialization of RDF, called Notation3.
TODO: provide a bit of EARL whereby people can just fill in the gaps; a bit like the bookmarklet thing.
<rdf:RDF xmlns="http://www.w3.org/2001/03/earl/1.00#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<Assertor rdf:parseType="Resource">
<name>NAME</name>
<email rdf:resource="mailto:EMAIL"/>
<asserts rdf:parseType="Resource">
<rdf:subject rdf:parseType="Resource">
<rdf:type
rdf:resource="http://www.w3.org/2001/03/earl/1.00#ENUM"/>
<!-- ENUM one of: WebContent/Tool/UserAgent -->
<testSubject rdf:resource="URI"/>
<date>DATE</date>
<!-- (or <version>VERSION<version>) -->
</rdf:subject>
<rdf:predicate
rdf:resource="http://www.w3.org/2001/03/earl/1.00#ENUM"/>
<!-- ENUM one of: passes/fails/notApplicableTo/notTestedAgainst/cannotTell -->
<rdf:object rdf:parseType="Resource">
<testCriteria rdf:parseType="Resource">
<id rdf:resource="URI"/>
</testCriteria>
</rdf:object>
</asserts>
</Assertor>
</rdf:RDF>
Here we should define a set of schemata that constitute a valid subset of XML RDF, and the EARL 1.0 model.
Here is a brief rundown of the terms in the EARL namespace:-
The EARL vocabularly, and to some extent the EARL model, are extensible; that is, they allow to to add new terms or otherwise modify them to fit your own specific application demands more closely. Indeed, the level of specificity in EARL is often not sufficent enough to avoid extending EARL. Thus, EARL was built to be extended. You may think of it as a core set of structures and terms.
For example, you may want to create your own ResultProperty, which has a custom level of confidence set. Let's say that you want to make an assertion that your page passes a certain checkpoint with a "medium" degree of confidence. To do that, you would have to create a new instance of ResultProperty, with a validity arc pointing to earl:Pass, and an confidence arc pointing to the new term "Medium". viz.:-
:myNewProperty rdf:type earl:ResultProperty; earl:validity earl:Pass; earl:confidence :Medium .
TODO: the set of result properties is actually explicitly enumerated, so we would have to change it to allow this.
Note that when you create a new property, you have to give it a name (this could be automatically generated), and put it in your own namespace. Do not under any circumstances invent new properties in the EARL namespace that don't exist.
You can then use the new property as normal:-
:Someone earl:asserts _:x . _:x rdf:subject :SomePage . _:x rdf:predicate :myNewProperty . _:x rdf:object :SomeCriteria .
TODO: How would EARL agents handle this? If they were fully RDF/SW-ized, then they'd be able to handle it just fine... but they won't be. We could perhaps offer CWM with EARL filters as a Web service, but that would introduce a certain amount of centralization. Still, deduct and then spit out the EARL 1.0 sounds quite useful.
@prefix : <#> .
@prefix earl: <http://www.w3.org/2001/03/earl/1.00-initial#> .
@prefix earldata: <http://www.w3.org/2001/03/earl/0.95datatypes#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix daml: <http://www.daml.org/2001/03/daml+oil#> .
@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix doc: <http://www.w3.org/2000/10/swap/pim/doc#> .
@prefix rcs: <http://www.w3.org/2001/03swell/rcs#> .
<> dc:title "EARL (Evaluation And Report Language) 1.0 RDF Schema";
doc:obsoletes <http://www.w3.org/2001/03/earl/0.95>;
doc:derivedFrom <http://www.w3.org/2001/03/earl/0.95>;
rcs:id "$Id$" .
earl:Evaluation rdfs:subClassOf rdf:Statement;
daml:intersectionOf
([ daml:onProperty rdf:subject; daml:toClass earl:Assertor ]
[ daml:onProperty rdf:predicate; daml:hasValue earl:asserts ]
[ daml:onProperty rdf:object; daml:toClass earl:Assertion ]);
rdfs:comment "An Evaluation is the most basic unit of EARL as it stands" .
earl:Assertor rdfs:subClassOf
[ daml:onProperty earl:platform; daml:toClass earl:TestPlatform ],
[ daml:onProperty earl:asserts; daml:toClass earl:Assertion ],
[ daml:disjointUnionOf (earl:Person earl:Machine) ];
rdfs:comment """The entity making the assertion (e.g. running the test). This
could be anonymous if for example you want to declare other
context information.""";
earl:MachineAssertor rdfs:comment "An assertor which is a machine";
daml:intersectionOf (earl:Assertor earl:Machine) .
earl:PersonAssertor rdfs:comment "An assertor which is a person";
daml:intersectionOf (earl:Assertor earl:Person) .
earl:asserts rdfs:comment """For earl:asserts(x,y), in general, the assertor (x)
asserts the assertion (y). EARL has specific
classes for these""" .
earl:Assertion rdfs:subClassOf rdf:Statement;
daml:intersectionOf
([ daml:onProperty rdf:subject; daml:toClass earl:TestSubject ]
[ daml:onProperty rdf:predicate; daml:toClass earl:ResultProperty ]
[ daml:onProperty rdf:object; daml:toClass earl:TestCase ]);
rdfs:comment """These could be free standing units - used outside of
earl:Evaluation(s) if required""" .
# earl:Assertor terms
earl:operator rdfs:domain earl:MachineAssertor;
rdfs:range earl:Operator . # who cares what an operator is? We won't define it
earl:contactInfo a rdf:Property;
rdfs:domain earl:Assertor; rdfs:range earl:ContactInfo .
earl:email a daml:UnambiguousProperty;
rdfs:subPropertyOf earl:contactInfo, foaf:mbox;
rdfs:domain earl:Assertor; rdfs:range earl:Email; # a machine can have one too
rdfs:comment
"""Generally, for earl:email(x, y) x has an email address of y.""" .
earl:name a rdf:Property;
rdfs:domain earl:Assertor; rdfs:range rdfs:Literal;
rdfs:comment "The common proper name for some earl:Assertor" .
earl:platform a rdf:Property;
rdfs:comment """e.g. the platform on which the test(s) were run, or the
general platform on which something runs""" .
earl:TestPlatform rdfs:comment "The platform on which the test(s) were run" .
earl:Platform rdfs:comment "The general platform on which something runs" .
earl:os rdfs:label "Operating System"; rdfs:subPropertyOf earl:platform .
earl:ccppSetting rdfs:comment "Just a strawman property" .
# earl:TestSubject terms
earl:TestSubject a rdfs:Class;
rdfs:subClassOf
[ daml:onProperty earl:uniqueMarker; daml:minCardinality "1" ],
[ daml:onProperty earl:platform; daml:toClass earl:Platform ] .
earl:WebContent rdfs:subClassOf earl:TestSubject .
earl:Tool rdfs:subClassOf earl:TestSubject .
earl:UserAgent rdfs:subClassOf earl:TestSubject .
(earl:WebContent earl:Tool earl:UserAgent) a daml:Disjoint .
earl:date a daml:UnambiguousProperty .
earl:version a daml:UnambiguousProperty .
earl:date a daml:DatatypeProperty; rdfs:range earldata:Date .
earl:testSubject a rdf:Property; # a daml:UniqueProperty;
rdfs:domain earl:TestSubject; rdfs:range rdfs:Resource .
# _:TestSubject
# earl:testSubject <http://example.org/>, <http://example.org/anotherPage/> .
earl:sameSubjectAs rdfs:seeAlso daml:equivalentTo;
rdfs:domain earl:TestSubject; rdf:range earl:TestSubject;
rdfs:comment """For earl:sameSubjectAs(x, y), x is defined as being the same
subject being evaluated (in an assertion) as y. In other
words, there is some property of sameness between the two -
the exact semantics of which are not defined here, but may
be useful to individual processors.""";
daml:inverseOf earl:sameSubjectAs .
earl:snapshot rdfs:domain earl:TestSubject .
earl:version a rdf:Property;
rdfs:comment "For earl:version(x, y), read: x is of version y" .
# earl:ResultProperty terms
earl:ResultProperty rdfs:subClassOf rdf:Property .
earl:validity rdfs:range earl:ValidityState;
rdfs:domain earl:ResultProperty .
earl:ResultProperty rdfs:subClassOf
[ daml:onProperty earl:note; daml:toClass rdfs:Resource ] .
# e.g.
earl:passes a earl:ResultProperty; earl:validity earl:Pass .
earl:fails a earl:ResultProperty; earl:validity earl:Fail .
earl:notApplicableTo a earl:ResultProperty; earl:validity earl:NotApplicable .
earl:notTestedAgainst a earl:ResultProperty; earl:validity earl:NotTested .
earl:cannotTell a earl:ResultProperty; earl:validity earl:CannotTell .
earl:ValidityProperty
daml:oneOf (earl:passes earl:fails earl:notApplicableTo
earl:notTestedAgainst earl:cannotTell) .
earl:confidence a rdf:Property . # kinda redundant
# earl:Assertion terms
earl:testMode a rdf:Property;
rdfs:domain earl:Assertion; rdfs:range earl:TestMode .
earl:Manual a earl:TestMode .
earl:Auto a earl:TestMode .
earl:Heuristic a earl:TestMode .
# earl:TestCase terms
earl:TestCase a rdfs:Class .
earl:repairInfo
rdfs:range earl:RepairInfo .
earl:RepairInfo rdfs:subClassOf
[ daml:onProperty earl:expectedResult; daml:toClass earl:ExpectedResult ] .
earl:purpose
rdfs:domain earl:TestCase; rdfs:range rdfs:Literal .
earl:testCriteria
rdfs:domain earl:TestCase;
rdfs:range earl:TestCriteria .
earl:TestCriteria daml:unionOf
([ daml:onProperty earl:level; daml:minCardinality "1" ]
[ daml:onProperty earl:excludes; daml:minCardinality "1" ]) .
earl:suite
rdfs:domain
[ daml:disjointUnionOf
(earl:TestCase earl:TestCriteria earl:Exclusion) ];
rdfs:range earl:Suite;
rdfs:comment """A suite of checkpoints, guidelines, or any other form of
rules that an earl:TestSubject can be evaluated against.
e.g. WCAG 1.0""" .
earl:id
rdfs:domain
[ daml:disjointUnionOf
(earl:TestCase earl:TestCriteria earl:Exclusion) ];
rdfs:range earl:Id; rdfs:comment "A single guideline, checkpoint, etc." .
earl:Suite daml:disjointWith earl:Id .
earl:excludes
rdfs:domain earl:TestCriteria;
rdfs:range earl:Exclusion .
earl:level
rdfs:domain
[ daml:disjointUnionOf (earl:TestCriteria earl:Exclusion) ];
rdfs:comment """This is a generic level of test criteria, for example,
the priorities/levels in WCAG. These shouldn't be a
string literal""" .
earl:operatorInstructions
rdfs:domain earl:TestCase; rdfs:range rdfs:Literal .
earl:reproducableStep
rdfs:domain earl:TestCase; rdfs:range rdfs:Literal .
earl:EARL rdfs:label "EARL (Evaluation And Report Language)";
earl:version "1.0" .
The working members of the ER WG, and important contributors to EARL at some point during its development (whether they knew it or not) include: Harvey Bingham, Dan Brickley, Wendy A. Chisholm, Dan Connolly, Daniel Dardailler, Al Gilman, Len Kasday, Nick Kew, Jim Ley, William "Geeze" Loughborough, Charles "Chaals" McCathieNeville, Libby Miller, Dave Pawson, Chris Ridpath, and Aaron Swartz.
Space does not permit listing of the folk whom contributed to RDF, XML, the W3C infrastructure, and all of the other things that we have depended upon, but we thank them anyway.
Sean B. Palmer $Id$