Comments on EARL 1.0 LC WD

Hello,

Here are a few comments on the Last Call Working Draft of "Evaluation
and Report Language (EARL) 1.0 Schema", published on March 23 2007 as a
Last Call Working Draft [1]:

RDF vs XML
----------
EARL is an RDF vocabulary which I think is a good thing, but is often
described as an XML one (e.g. the way the contraints are set on
elements/properties; see also the comment on RFC keywords below).

I think there ought to be two separate formal specifications (not
necessarily in separate documents, though):
* one to express constraints that are true in a RDF world (in term of
OWL constraints and semantics)
* one to express constraints on a subset of EARL, that would be enforced
at the XML level (e.g. through an XML schema, or any other schema
language), and that would basically define the set of data a given class
os product (typically, testing software) should be expected to output,
within a certain well-defined set of XML constraints - this would
certainly mean reducing the flexibility of the syntax that EARL-as-RDF
can be expressed through

The advantage of having both is to have on one hand a formal
semantically-rich definition of what EARL describes, and on the other
hand an RDF-compatible XML version of the data that is more likely to be
parseable through a greater number of tools, with additional semantic
restrictions (e.g. that all the required data be available in a single
XML document vs the "open world" hypothesis behind RDF/OWL).

Use of RFC 2119 Keywords
------------------------
The document uses the RFC keywords to put constraints on the language
constructions, e.g.:
        "An Assertion must have exactly one instance of each of the
        following properties"
As discussed in the QA Wiki [2], these keywords should preferably be
used for constraining implementations rather than the language itself;
this is all the more true given how the constraints are expressed
semantically: essentially, the OWL ontology says:
"an Assertion has exactly one instance of each of the following
properties" (i.e. it is not that it "must" have exactly one, but that if
more than one is known, that means the two have the same objects, and if
none is known, it just means that it is yet to be known)

Thus, I would recommend switching to the declarative form for the
statements of this sort ("must have" => "has").

Compound Assertor
-----------------
It would be useful to clarify whether either of the following is true:
        if a given test is determined to pass by a compound assertor,
        then each of the sub entities of the compound assertor asserts
        that the test passes
        vs
        binding the determination of whether a given test passes to a
        compound assertor only asserts that the entities taken as a
        whole asserts that the test passes
        
Test Modes
----------
The names used in the classification of the test modes seem a bit weird:
* "manual" is used to refer to something "based on a person's judgment";
this reads as "subjective" to me, rather than manual; I don't know if
the term or the definition is wrong; also, this refers to "a person",
but does that include organizations as well? (if so, it should probably
refer to an agent instead; if not, it should be made explicit)
* "heuristic" reads to me as something that was not 100% surely
determined, which is quite different from what the current definition
reads; again, I'm not sure which of the term or the definition is wrong
* "notAvailable" is not a great name for what is meant either; I would
think "undetermined" (or maybe "undeterminedMode" for sake of clarity)
would be closer to what is meant

Outcome Value
-------------
* While I think the 5 defined classes are flexible enough for most
conformance testing, I'm not sure there is any reason to restrict
earl:outcome to these 5 classes is necessary; it prevents to re-use EARL
in other contexts (e.g. performance testing), for no good reason that I
can see (except over-constraining the RDF for XML-reasons, see comment
RDF vs XML)
* "pre-defined values" should read "pre-defined classes"

Range mixing Literals and Resources
-----------------------------------
earl:sourceCopy can take both Literals and URI-s identified resources as
objects; that seems wrong to me, but I couldn't find anything
discouraging it explicitly; you may want to ask an RDF-modeling expert
to check if it's ok, though.

earl:Content
------------
* How does earl:Content relates to the notion of "Information Resources"
as defined by the TAG?
http://www.w3.org/TR/2004/REC-webarch-20041215/#def-information-resource
It would be useful to clarify whether Content is a subset of
InformationResource, or the same thing (and if so, it would probably be
better to re-use an existing vocabulary that describes it).
* earl:Content mentions only HTTP resources; that excludes HTTPS, but
also FTP, etc; it may be more appropriate to have a generic earl:Content
class that doesn't restrict the specific access protocol, and have a
subclass earl:HTTPContent that allows to identify the well-known case of
content accessed through HTTP


RDF Schema
----------
* the rfds:comment for Assertion reads "Parent node that contains all
parts of an assertion"; this is a syntactic-description (and an
XML-centered one too) instead of a semantic one that would be expected
in an RDF schema
* the description of assertor and single assertor mentions "person" but
not organizations
* earl:Content only mentions "on the Web", while the textual definition
mentions availability through HTTP

Editorial bugs
--------------
* the copyright reads "2007" while the spec was started much earlier
than that
* The table of content has two "2.2.1" (the second one should obviously
be "2.2.2")
* http://www.w3.org/TR/2007/WD-EARL10-Schema-20070323/#compoundassertor
reads "these instances can be a Person, Agent, Software, or recursively
another Compound Assertor"; I think it should either read "a Person,
Organization, Software, or...", or "an Agent, Software, or ..." (i.e.
it's not clear why organization doesn't appear while person does)


HTH,

Dom

1. http://www.w3.org/TR/2007/WD-EARL10-Schema-20070323/
2. http://esw.w3.org/topic/RfcKeywords

Received on Wednesday, 11 April 2007 19:13:08 UTC