earl (vocab) design issues -- rough notes

I just wrote this up, instead of trying to say it verbally. Not sure if
it helps.... --dan




EARL design issues  -- rough notes

danbri@w3.org


Overview

EARL is concerned with capturing and exchanging accurate records of
test results, while preserving knowledge about 'who said what' and
highlighting
areas of agreement and disagreement.

EARL descriptions are written in XML according to the RDF data
modelling conventions. An EARL XML/RDF document can be read as
making claims that might be true or false of the world. Those claims are
themselves often about other (more controversial) claims about the state
of the world, eg. that some tool or content fails (or passes) some test.
As a consequence, EARL has some particularly demanding technical
requirements.

The claims made by each EARL document are typically about other, more
controversial, claims that various parties (human or machine) are making.
This
indirectness makes it possible for consumers of EARL data (tools,
databases)
to merge information from multiple sources to produce an
aggregated account of "who said what" regarding tests, and the
success or failure details relating to the application of various tests
to content, tools etc.

EARL documents are themselves expected to make pretty uncontroversial
claims,
such as 'on tuesday, Bill checked document a.html
against test case t14, and claims that it failed; on wednesday, Alice
checked tool tool123 against test t213 and claims that it passed'.

The claims in these EARL descriptions are generally passive, neutral, and
not
particularly error prone or debatable. We might make mistakes, of course,
but we'll
in most cases not have much dispute about the claims directly made by EARL
documents.

We can think about these 'top level' claims as the sort of statements
that we might preface with 'everone agrees that...'.
Everyone might agree that 'Chris tested b.html on wednesday', while still
disagreeing with Chris's resulting claim that 'b.html fails test t432' or
with
the reasons he gives. The EARL design is focussed on characterising such
agreements
and disagreements in a way that allows for the exchange of a reasonably
simple
data format which nevertheless treats areas of controversy with some care.


EARL claim types

We can distinguish between two kinds of claims.

(i)
Those that are considered 'top level' and uncontroversial in EARL.

Such claims should be  reasonably safe for merging by aggregating multiple
EARL documents.
For example:  'dan tested some piece of webcontent and claims that it
failed'.

(ii)
Those that are embedded or 'scare quoted' in EARL, and potentially
controversial.

Such claims need to be associated with additional information, such as who
made them,
what content/tool/etc they are being made of, etc etc. An example
controversial claim
might be that 'tool tool-323 fails test case t12'.

Both kinds of claim can be written in XML and modelled as RDF statements,
ie. using
the RDF graph representation.

The normal model for EARL is to package up controversial claims (eg. about
success or failure
of tools/content w.r.t. test cases). These controversial claims are, from
EARL's point of
view, re-characterised in a quoted form.

Problem Areas / Discussion
--------------------------

(i) unexpected disagreement

Claims that we might be modelling as if they were uncontroversial, might
turn out to be
areas of possible (unanticipated, accidental...) dispute.

Example:

Two EARL descriptions talk about some TestSubject and ascribe different,
conflicting
properties to it. For example: sizeInBytes, sha1sum, md5sum etc.

see http://www.ecs.soton.ac.uk/~nmg/er-swad/earl-frag2.xml test case.

(ii) controversial extensions

EARL currently hides/protects only a portion of its content, basically
just the
tiny bit that is clearly controversial: whether or not the test subject
fails
some test.

tentative conclusions:
Folk adding extension properties into EARL descriptions have
responsibility for making
these new claims uncontroversial. If EARL descriptions are extended in
ways that make
it easy for test descriptions to disagree, we have a problem.

We all understand that diverse EARL documents might describe test results
that disagree;
the important thing is that we have an over-arching framework for
describing such disagreements.

This is an important part of clarifying the partionioning within EARL
between
controversial and uncontroversial claims. There is generally a high
overhead (technical,
database/query complexity, file format verbosity) associated with putting
information in the
controversial category.

NOTES from talking in meeting:

Jim: this is revisiting design discussions from earlier this year. The
idea of
having URIs on the test subject was rejected for much this reason. We're
now using
bNodes; the URIs on test subject case is a strawman.

[d: see IE CSS tests for example of doing this]

Danbri: there's still a problem. We need to provide careful guidance for
those extending
EARL. If folk add in new properties to their test subject node, and those
properties
are unambiguous properties, we'll get an unwanted effect: smart RDF stores
will fold
together these nodes.

We want the things that are test subjects to be uncontroversial, ie. there
be few/no
circumstances where we can reasonably (...) disagree about their
characteristics.

The current EARL design uses bNodes here, to help ensure that mixed/merged
descriptions don't
mangle together properties from disagreeing descriptions of the things
these nodes represent.
(eg. web pages, tools).

In addition, we need to offer guidance on properties and types usable in
the test subject
role in an EARL description, constraining these so that EARL descriptions
can be safely merged.

We can do this by ensuring that this node always stands for a static,
snapshotted or
timeless view of a resource.

In the case of a Web document, this might be the kind of thing that
reasonably has a
single, widely agree-able filesize, checksum etc.

In the case of a Tool, eg. IE version v3.0342341 it might be a highly
specific version of
some software.

The specific software might be relatable to a more generic notion of that
thing, eg. by
an earl:genericVersion property or other domain-specific modelling
vocabulary.
Similarly a document instance/snapshot might be related to a generic
abstract work. The latter
(ie. the abstract document, or the software tool abstractly conceived)
could quite
safely be named by URI or related by an unambiguous property to some URI.
In the former case,
webdocs (considered as changeable works) could be given traditional URIs,
in the latter,
we might for eg record the 'homepage' of some software.

This modeling idiom allows the best of both worlds. We could query for
test subjects that were
specific versions of some easily-queried-on pattern ('the piece of
software with homepage foo';
'the doc with uri bar') that picks out the thing 'in the abstract'.

EARL authors, and extension designers, need to take care not to plug in
controversial
property/value assertions, either at the level of the specific resource
(test subject) or,
if represented, the generic resource it is a version or snapshot of.

This constrains the EARL schema design and the design of extensions to the
EARL description
framework. If folk mix things up, they'll cause problems for EARL
aggregators.

Basic rule of thumb:

 * if you're saying something in your EARL RDF that ascribes properties to
the
   resource indicated by the test subject node, only say things you
believe are
   generally and timelessly true. The associated schema should be designed
to
   compliment this: eg. WebContent might be understood as
WebContentSnapshot...


...rambling notes end here so I can pay attention in meeting (we're
talking
about XSLT to convert between verbose/paranoid and simple/lossy
formats...)

--
danbri@w3.org

Received on Tuesday, 25 June 2002 09:45:29 UTC