Copyright © 2002 W3C® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
This is a W3C Working Draft produced by the Evaluation and Repair Tools Working Group (ERT WG). The purpose of this document is to explain how and why to use the Evaluation and Report Language (EARL) 1.0. The Working Group encourages feedback about this document as well as implementation of the language in authoring tools, testing tools, search engines, and other relevant tools.
The Evaluation and Report Language (EARL) is a general-purpose language for representing conformance information on the Web. This specification describes how to use EARL to describe conformance and defines a basic vocabulary for this purpose.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.
This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress". A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.
Send comments about this document to the Evaluation and Repair Tools Working Group. The archives for this list are publicly available.
This document has been produced as part of the W3C Web Accessibility Initiative (WAI). The goals of the ERT WG are discussed in the Working Group charter. The ERT WG is part of the WAI Technical Activity.
When creating content to publish on the Web a developer runs a variety of tests:
If working on many pages, there can be a variety of errors to test further, fix, and republish.
Developers of authoring tools and user agents also test their software for bugs, conformance to specifications, accessibility, and usability.
EARL is a format for describing the results of these tests.
@@define:
@@ include stuff from:
Scenario: A consultant evaluting a client's Web site. Uses a variety of evaluation tools to generate an accessibility report of the site.
@@describe when EARL is generated, where it is stored, how it is used (for each scenario).
Scenario: A tester evaluating a company's Web site. Uses a variety of evaluation tools to discover possible bugs on the site.
"power developer" that uses programming tools to produce site, not a WYSIWYG editor commonly used by less technical folk. QA folk use different tools than developers. (@@Jenae Andershonis would be good reviewer for this scenario)
Scenario: A developer maintaining a company's Web site. Fixes bug reports generated by the tester.
Scenario: A W3C Working Group is trying to meet their exit criteria for Candidate Recommendation. To do this they need to show at least two indepdenent implementations.
@@refering to test suites.
Scenario: A person uses a Web search engine to find specific information. They are using a screen reader and want the results sorted with the accessible items listed first.
Scenario: A person provides technical support for an assistive technology. As the assistive technology runs on the user's machine, it logs when it runs into trouble. The user sends this log to technical support. Technical support is better able to assist the user.
Scenario: A variety of people test the same tool. They compare their results to discover where they agree and disagree and to come to a final result.
/* brief intro to RDF class/property model */
The class of all EARL evaluations, as asserted in an EARL document. A sub class of rdf:Statement. (An Evaluation is the most basic unit of EARL as it stands. An EARL document will always have at least one Evaluation asserted in it.)
An assertion is a sub class of statement, where the subject is a TestSubject, the predicate a ResultProperty, and the object a TestCase. These could be free standing units - used outside of Evaluation(s) if required. N.B. Assertion is the domain of earl:assertedBy.
The class of entities making an assertion in EARL (e.g. running the test). This could be anonymous if required. There are two further sub classes of Assertor defined, but they are basically unused. An Assertor can be a Person or a Machine (but not both).
The class of things which are being evaluated in an EARL evaluation - the things that are the subject of every assertion. This needs to be qualified with some type of information in order to make it unambiguous. You may use an unambiguous property, or unambiguous constellation of properties.
This is a tool; a bit of software, perhaps with a desc.
A user agent
WebContent is synonymous with "Document Source" in UAAG, and "Representation" in RFC 2616. Anything which is an instance of WebContent must have exactly one reprOf predicate hanging from it.
The set of result properties in EARL; they are the range of rdf:predicate for all EARL Assertions.
The class of all TestCases. Conceptually, a TestCase is some resource to which another resource can be validated against - something for which it can be demonstrated that it passes or fails against (or otherwise). This may in fact include many things - validation classes, code test cases, or more subjective guidelines such as WCAG. Perhaps this is synonymous with "Class", and we're really just testing class membership? For example, validation is a test of whether a set of characters is in a particular document class. Testing against WCAG is testing for whether or not a document is in the class of accessible documents. Testing against a test file checks whether the code belongs to the "Pass" class created by the test file.
For earl:assertedBy(y,x), in general, the assertor (x) asserts the assertion (y). EARL has specific classes for the domain/range
The test mode of some EARL assertion
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#Assertion
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#TestMode
@@is TestMode a subclass of something else?
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#Assertor
The range for this property is: http://www.w3.org/2000/01/rdf-schema#Literal
For contactInfo(x, y), y is some some kind of generic contact information for y. This is a sub classable property. A strict domain is used rather than a restriction so that one may infer that the subject is an Assertor. Not that that information would actually help.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#Assertor
The domain of this property is:
The Operating System on which an assertion was run, or of which an Assertor uses.
For platform(x, y), y is the platform on which x ran/runs. E.g. the platform on which the test(s) were run, or the general platform on which something runs
The MIME type of the WebContent subject, as a literal.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#WebContent
The range for this property is: http://www.w3.org/2000/01/rdf-schema#Literal
For reprOf(x, y), read that x is a representation of y, possibly derived from an HTTP GET transaction.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#WebContent
The range for this property is: http://www.w3.org/2000/01/rdf-schema#Resource
Points to a snapshot of a serialization of the resource as it was when it was evaluated.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#TestSubject
The range for this property is: http://www.w3.org/DesignIssues/Generic#Fixed
for confidence(x, y), x has the confidence y
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#ResultProperty
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#ConfidenceLevel
How confident is the result of the test? Are you certain it fails? Uncertain that it passes?
The validity state for an EARL ResultProperty.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#ResultProperty
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#Validity
The class of all EARL validity states.
for excludes(x, y), TestCriteria x exclude the Exclusion y
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#TestCriteria
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#Exclusion
A single guideline, checkpoint, etc.
The domain of this property is:
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#Id
This is a generic level of test criteria, for example, the priorities/levels in WCAG. These shouldn't be a string literal.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#Exclusion
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#TestCriteria
The expected result of a test case
For operatorInstruction(x, y), y is an operator instruction for x.
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#TestCase
The range for this property is: http://www.w3.org/2000/01/rdf-schema#Literal
For purpose(x, y), y is the (literal node) purpose of x
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#TestCase
The range for this property is: http://www.w3.org/2000/01/rdf-schema#Literal
for repairInfo(x, y), y is repair information for x
A suite of checkpoints, guidelines, or any other form of rules that an earl:TestSubject can be evaluated against. e.g. WCAG 1.0
The domain of this property is:
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#Suite
The testCriteria for a TestCase
The domain of this property is: http://www.w3.org/2001/03/earl/1.0-test#TestCase
The range for this property is: http://www.w3.org/2001/03/earl/1.0-test#TestCriteria
| properties | range of property | instances of | constraints | comment |
|---|---|---|---|---|
| object | TestCase | |||
| subject | TestSubject | reprOf, UserAgent, Tool | ||
| predicate | ResultProperty | cannotTell, fails, notApplicableTo | ||
| assertedBy | Assertor | Assertor can be a Person or Machine but not both | . | |
| testMode | TestMode | The test mode of some EARL assertion |
| properties | range of property | comment |
|---|---|---|
| id | Id | |
| suite | Suite | suite of checkpoints/rules to evaluate against |
| testCriteria | TestCriteria | test criteria for a TestCase |
| operatorInstruction | Literal | For operatorInstruction(x, y), y is an operator instruction for x. |
| purpose | Literal | For purpose(x, y), y is the (literal node) purpose of x |
| properties | range of property | comment |
|---|---|---|
| confidence | ConfidenceLevel | for confidence(x, y), x has the confidence y |
| validity | Validity | The validity state for an EARL ResultProperty. |
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
| contactInfo | For contactInfo(x, y), y is some some kind of generic contact information for y. This is a sub classable property. A strict domain is used rather than a restriction so that one may infer that the subject is an Assertor. Not that that information would actually help. | ||
| Generally, for earl:email(x, y) x has an email address of y. The URI of the object will normally start with "mailto:", and will not be a literal. This is not a sub property of foaf:mbox! That property can only be applied to people. | contactInfo | ||
| name | Literal | The common proper name for some earl:Assertor. | foaf:name |
| operator | for operator(x, y), y is the operator of x | Assertor |
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
| excludes | Exclusion | for excludes(x, y), TestCriteria x exclude the Exclusion y | |
| id | Id | ||
| suite | Suite | suite of checkpoints/rules to evaluate against | |
| level | This is a generic level of test criteria, for example, the priorities/levels in WCAG. These shouldn't be a string literal. |
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
| format | Literal | The MIME type of the WebContent subject, as a literal. | |
| reprOf | rdfs:Resource | For reprOf(x, y), read that x is a representation of y, possibly derived from an HTTP GET transaction. | |
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
| id | Id | ||
| suite | Suite | suite of checkpoints/rules to evaluate against | |
| level | This is a generic level of test criteria, for example, the priorities/levels in WCAG. These shouldn't be a string literal. | ||
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
| id | Id | ||
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
| snapshot | DesignIssues:Fixed | Points to a snapshot of a serialization of the resource as it was when it was evaluated. | |
| properties | range of property | comment | subPropertyOf |
|---|---|---|---|
@@describe how EARL can be extended. Use PageValet and HiSoft as examples? Create others?
@@for each user scenario described above, create a sample piece of EARL.
User Scenarios