Re: Questions and proposals related to strawman schema from F2F

Nick,

Thanks for the comments.  I have a few follow-up questions.

My response is delayed from taking a holiday then catching up from being on 
holiday.

>Wendy A Chisholm <wendy@w3.org> writes:
> > 1. Severity (new)
>
> > At the F2F we had consensus that a severity property, similar to
> > confidence and validity, would be a good thing.  I propose we add
> > something to the strawman.

Nick Gibbons replied:
>As given in Ian's description, severity doesn't seem to be orthogonal
>to confidence and validity. The meaning of a severity value therefore
>depends on the validity of the test (a high severity pass is 'better'
>than a low severity pass, but a low severity fail is 'better' than a
>high severity fail). This leads to a situation where there are some
>combinations of severity and validity whose combination seems
>ill-defined: what meaning should be ascribed to severity for the
>validity values notTested, notApplicable and cannotTell?
>
>This feature can be added to the schema with little effort, but I
>think that we need to be sure about exactly what it means before we do
>that.

Ian, from your message it seems that severity is only intended to be used 
with pass or fail and not with notTested, notApplicable, and 
cannotTell.  Can you confirm this?

> > 4.  testMode (from 1.0 test)
>
> > testMode had 3 instances: automatic, heuristic, and manual. [...]
> > If not covered by Assertor properties, I propose we add something to
> > the strawman.
>
>There are two separate issues here. sbp's 1.0 proposal had both a test
>mode and subclasses of Assertor (MachineAssertor and PersonAssertor).
>If the mode of an assertion is automatic, should it be understood that
>the assertor is a machine (or conversely, if it is manual, is the
>assertor a person)? Is a manual assertion made by a machine
>nonsensical?

Good questions.  I think automatic assertions are always made by machine 
assertors, but I don't think the converse is always true. I think it is 
possible for a machine to make a manual assertion.  e.g. several 
accessibility evaluation tools prompt the user to perform manual 
tests.   The tool could assert the test result and that it was derived from 
human input.  This should also increase the confidence of the assertion.

>I suspect that EARL needs only one of the two ways of indicating how a
>test was performed; subclassing Assertor seems intuitively simpler,
>but explicitly indicating the mode would also work adequately.

It seems to me that it would make more sense to have a property of the 
assertion rather than subclass the assertor.  I think the more interesting 
information is how the testResult is derived rather than knowing more about 
the assertor.  I realize that more information about the assertor generally 
increases the trust that people have in the assertion,  but I think in this 
case the test mode is a property of the assertion rather than the assertor.

>The second issue concerns the meaning of the heuristic mode. I've not
>been able to find a concrete definition for this, so it is difficult
>to say whether or not it will be covered by Assertor properties.

If I remember correctly:
* automatic - the test is cut and dry.  e.g. either the img element has an 
alt attribute or it does not
* heuristic - we can check for patterns but this might not be exhaustive or 
it might generate a false positive.  e.g. the alt-text of this image seems 
to be ok since it does not contain the following suspicious phrases: 
".gif", "insert alt-text here", "KB", "image", etc.
* manual - there are no checks that the machine performs, although it can 
give the user some idea of where to look for potential problems.  e.g. the 
script element is present.

I haven't seen heuristic used too much and it might not be worth 
keeping.  The lines are fairly blurry.  We could define automatic and 
manual as follows:
automatic: the test was run without human intervention.
manual: the test required human intervention.

Thoughts?

> > 5. TestCriteria,  suite, level, excludes (from 1.0 test)
>
> > While these describe pieces of the testCase, at the face to face we
> > agreed that we need some way to group tests.  Suite, level, and
> > excludes were created to do this.  I propose we add something to the
> > strawman.  Ideas?
>
>I think that EARL should distinguish between groups of testcases and
>groups of the criteria that the testcases test.

agreed.

>The line between a testcase and a criterion is also a bit blurry - are
>WCAG checkpoints considered to be testcases or criteria?

In WCAG 2.0 we have 3 levels:
1. Guidelines
2. Checkpoints
3. Success criteria

a success criterion is the child of one checkpoint and the great-grandchild 
of one guideline.
a checkpoint is the child of one guideline and at least one child (success 
criterion).
a guideline has siblings (other guidelines) and more than one child 
(checkpoint).
the root is the document "WCAG 2.0"

Although, people won't be claiming conformance to Guidelines, they will 
only be testing against checkpoints and success criteria.  Thus, I think 
your proposed schema would work for WCAG 2.0 conformance testing. (I tried 
it out below...really quick first pass...)

CSS Test Suite seems to group more than one test under one title.  e.g. 2.1 
"anchor" [1] seems to contain 5 tests.  Or are these considered one test?
A {color: green;}
A:link {color: purple;}
A:visited {color: lime;}
A:active {color: maroon;}
#one {color: #006600;}

[1] http://www.w3.org/Style/CSS/Test/CSS1/current/sec21.htm

Using the "throwaway strawman" from [2]
http://lists.w3.org/Archives/Public/w3c-wai-er-ig/2002Aug/0009.html

We could come up with the following.  I'm not sure how I feel about 
this.  I didn't need to use Criterion, CriterionGroup, containsCriterion, 
or tests.  I really like the idea of tests, tho...
png of the graph is at: 
http://lists.w3.org/Archives/Public/www-archive/2002Sep/0110.html

<rdf:RDF xmlns="http://www.w3.org/2001/03/earl/0.95#"
     xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
     xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
  <Assertor>
   <name>Bob B. Bobbington</name>
   <email rdf:resource="mailto:bob@example.org"/>
   <asserts>
     <Assertion>
       <subject rdf:resource="http://www.w3.org"/>
      <result rdf:resource="http://www.w3.org/2001/03/earl/0.95#passes"/>
       <testcase rdf:resource="http://example.org/#WCAG20-Checkpoint-1"/>
     </Assertion>
   </asserts>
  </Assertor>

  <TestGroup rdf:about="http://example.org/#WCAG20-Checkpoint-1">
     <containsTestCase rdf:resource="http://example.org/#success-criterion-1"/>
     <containsTestCase rdf:resource="http://example.org/#success-criterion-2"/>
  </TestGroup>
  </rdf:RDF>

>Excluded from this are, well, exclusions. I couldn't find any clear
>indication of how these are expected to be used - examples would be
>welcome.

Yes, I'm not sure how to use exclusions either.  I think we can drop it.


> > 6. os, version (from 1.0 test)
>
> > Is this covered by platform.  It seems to me that these are
> > necessary to uniquely identify user agent test subjects.  If not
> > covered by platform, I propose we add something to the strawman.
>
>In the strawman (and in sbp's proposal), platform describes aspects of
>the environment in which the assertor performed the tests, rather than
>aspects of the subject of the test. In sbp's proposal, os is a
>refinement of platform, and so also describes the test environment,
>not the test subject.
>
>In the strawman, user agent test subjects are identified by a URI,
>with a human-readable label being provided using rdfs:label. If it is
>necessary to explicitly state the OS and version of the UA
>(rather than implicitly coding this in the URI used to identify the
>UA), the schema can easily be modified to include os and version
>properties.

If each version and os release of a UA is identifiable by a unique URI then 
that should use a URI.  What is the URI for Opera v5 for Mac OS X? is it 
http://www.opera.com/download/get.pl?platform=mac&force=5.0 ?  That seems 
to be the only thing I could find on their site.

This is similar to identifying the reprof of web content with the 
normalization algorithm used and result.

Should we include earl properties to uniquely identify test subjects (i.e. 
normalization algorithm, hash algorithm, hash values for web content  and 
os, version for tools)?


> > 8. date  (from 1.0 test)
snip, snip
>For this reason, I believe that the right course to take for EARL 1.0
>on this issue would be to use the date property from the DC vocabulary
>in preference to creating a date property for use with EARL alone.

Sounds good to me.

--wendy

-- 
wendy a chisholm
world wide web consortium
web accessibility initiative
seattle, wa usa
/--

Received on Tuesday, 10 September 2002 18:18:13 UTC