Re: Questions and proposals related to strawman schema from F2F

Hi Wendy,

Lengthy comments follow below. If I've not replied to an issue you
raised, I'm in agreement with you.

Wendy A Chisholm <wendy@w3.org> writes:

> I compared the strawman to the 1.0 test schema [2].  What follows is
> a list of classes and properties not defined in the strawman and
> proposals for how to deal with them (or not). Assuming we move
> forward with the strawman, I want to be sure we don't accidentally
> leave out something important.

Agreed. The strawman I drew up was incomplete because I concentrated
on certain (more controversial) aspects of EARL and left out the less
controversial features in order to clarify my proposals.

> Please let me know if you agree or disagree with any of these
> proposals.
  
> 1. Severity (new)

> At the F2F we had consensus that a severity property, similar to
> confidence and validity, would be a good thing.  I propose we add
> something to the strawman.

As given in Ian's description, severity doesn't seem to be orthogonal
to confidence and validity. The meaning of a severity value therefore
depends on the validity of the test (a high severity pass is 'better'
than a low severity pass, but a low severity fail is 'better' than a
high severity fail). This leads to a situation where there are some
combinations of severity and validity whose combination seems
ill-defined: what meaning should be ascribed to severity for the
validity values notTested, notApplicable and cannotTell?

This feature can be added to the schema with little effort, but I
think that we need to be sure about exactly what it means before we do
that.

> 4.  testMode (from 1.0 test)

> testMode had 3 instances: automatic, heuristic, and manual. [...]
> If not covered by Assertor properties, I propose we add something to
> the strawman.

There are two separate issues here. sbp's 1.0 proposal had both a test
mode and subclasses of Assertor (MachineAssertor and PersonAssertor).
If the mode of an assertion is automatic, should it be understood that
the assertor is a machine (or conversely, if it is manual, is the
assertor a person)? Is a manual assertion made by a machine
nonsensical?

I suspect that EARL needs only one of the two ways of indicating how a
test was performed; subclassing Assertor seems intuitively simpler,
but explicitly indicating the mode would also work adequately.

The second issue concerns the meaning of the heuristic mode. I've not
been able to find a concrete definition for this, so it is difficult
to say whether or not it will be covered by Assertor properties.

> 5. TestCriteria,  suite, level, excludes (from 1.0 test)

> While these describe pieces of the testCase, at the face to face we
> agreed that we need some way to group tests.  Suite, level, and
> excludes were created to do this.  I propose we add something to the
> strawman.  Ideas?

I think that EARL should distinguish between groups of testcases and
groups of the criteria that the testcases test. sbp's schema combines
these both as Suite, which therefore encompasses groups of testcases,
groups of criteria and also mixed groups (I can't think of a likely
occurrence for moxed groups).

The line between a testcase and a criterion is also a bit blurry - are
WCAG checkpoints considered to be testcases or criteria?

As another throwaway strawman, consider the following
back-of-an-envelope RDF fragment:

<rdfs:Class rdf:about="&earl;TestCase"/>

<rdfs:Class rdf:about="&earl;TestGroup">
  <rdfs:comment>A group of testcases</rdfs:comment>
  <rdfs:subClassOf rdf:resource="&earl;TestCase"/>
</rdfs:Class>

<rdf:Property rdf:about="&earl;containsTestCase">
  <rdfs:comment>Relates a testcase group to the testcases it
    contains</rdfs:comment> 
  <rdfs:domain rdf:resource="&earl;TestGroup"/>
  <rdfs:range rdf:resource="&earl;TestCase"/>
</rdf:Property>

<rdfs:Class rdf:about="&earl;Criterion">
  <rdfs:comment>A checkpoint or criterion against which testcases
    are designed to assess conformance. Example: a WCAG
    checkpoint</rdfs:comment> 
</rdfs:Class>

<rdfs:Class rdf:about="&earl;CriterionGroup">
  <rdfs:comment>A collection of criteria, such as the WCAG
    guidelines</rdfs:comment> 
  <rdfs:subClassOf rdf:resource="&earl;Criterion"/>
</rdfs:Class>

<rdf:Property rdf:about="&earl;containsCriterion">
  <rdfs:comment></rdfs:comment>
  <rdfs:domain rdf:resource="&earl;CriterionGroup"/>
  <rdfs:range rdf:resource="&earl;Criterion"/>
</rdf:Property>

<rdf:Property rdf:about="&earl;tests">
  <rdfs:comment>Relates a testcase (or testcase group) to the criteria
    against which it is designed to assess conformance</rdfs:comment>
  <rdfs:domain rdf:resource="&earl;TestCase"/>
  <rdfs:range rdf:resource="&earl;Criterion"/>
</rdf:Property>

Note that there are two separate meronomic hierarchies, one for
testcases and one for criteria. It is trivially possible to create
groups of groups, which is one way of implementing conformance levels.

Excluded from this are, well, exclusions. I couldn't find any clear
indication of how these are expected to be used - examples would be
welcome.

> 6. os, version (from 1.0 test)

> Is this covered by platform.  It seems to me that these are
> necessary to uniquely identify user agent test subjects.  If not
> covered by platform, I propose we add something to the strawman.

In the strawman (and in sbp's proposal), platform describes aspects of
the environment in which the assertor performed the tests, rather than
aspects of the subject of the test. In sbp's proposal, os is a
refinement of platform, and so also describes the test environment,
not the test subject.

In the strawman, user agent test subjects are identified by a URI,
with a human-readable label being provided using rdfs:label. If it is
necessary to explicitly state the OS and version of the UA
(rather than implicitly coding this in the URI used to identify the
UA), the schema can easily be modified to include os and version
properties.

(note also that version is only used in sbp's schema to denote the
version of EARL itself)

> 8. date  (from 1.0 test)

> I believe SBP created an EARL-specific date datatype to deal with
> ambiguities in DC. Date is important in identifying "versions" of
> dynamic web content.  The strawman does not contain any constraints
> other than rdfs:range and rdfs:domain, whereas 0.95 and test1.0 had
> a variety of constraints (mostly daml).  Many folks felt that
> constraints was something to attempt in a future version after
> daml/oil/owl/other are stable.  Is creating a datatype for dates a
> similar issue?  Has this been dealt with someplace else (DC?) since
> earl:date was created?  I do not have a proposal one way or the
> other at this time.

From what I can tell, sbp writes dates in the format defined by
ISO8601 (see http://www.w3.org/TR/NOTE-datetime), as is also
recommended by DCMI for the value of the dc:date property.

The date property in sbp's schema has been defined as a
daml:DatatypeProperty with a range taken from another EARL-specific
schema (does this other schema exist? I couldn't find a copy of it).

While I appreciate that this design decision was made to constrain the
range of the property to something smaller than rdfs:Literal, I stand
by the comments I made at the f2f, namely that EARL 1.0 should not use
vocabulary from DAML+OIL, which is likely to be deprecated in favour
of OWL, or from OWL, which is not likely to reach stability within the
development lifetime of EARL 1.0.

For this reason, I believe that the right course to take for EARL 1.0
on this issue would be to use the date property from the DC vocabulary
in preference to creating a date property for use with EARL alone.

Any comments are welcome,
-- 
Nick Gibbins                                            nmg@ecs.soton.ac.uk
IAM (Intelligence, Agents, Multimedia)             tel: +44 (0) 23 80592831
Electronics and Computer Science                   fax: +44 (0) 23 80592865
University of Southampton

Received on Friday, 23 August 2002 11:40:38 UTC