- From: Wendy A Chisholm <wendy@w3.org>
- Date: Tue, 10 Sep 2002 18:23:22 -0400
- To: ian@hixie.ch, nmg@ecs.soton.ac.uk
- Cc: w3c-wai-er-ig@w3.org
Nick, Thanks for the comments. I have a few follow-up questions. My response is delayed from taking a holiday then catching up from being on holiday. >Wendy A Chisholm <wendy@w3.org> writes: > > 1. Severity (new) > > > At the F2F we had consensus that a severity property, similar to > > confidence and validity, would be a good thing. I propose we add > > something to the strawman. Nick Gibbons replied: >As given in Ian's description, severity doesn't seem to be orthogonal >to confidence and validity. The meaning of a severity value therefore >depends on the validity of the test (a high severity pass is 'better' >than a low severity pass, but a low severity fail is 'better' than a >high severity fail). This leads to a situation where there are some >combinations of severity and validity whose combination seems >ill-defined: what meaning should be ascribed to severity for the >validity values notTested, notApplicable and cannotTell? > >This feature can be added to the schema with little effort, but I >think that we need to be sure about exactly what it means before we do >that. Ian, from your message it seems that severity is only intended to be used with pass or fail and not with notTested, notApplicable, and cannotTell. Can you confirm this? > > 4. testMode (from 1.0 test) > > > testMode had 3 instances: automatic, heuristic, and manual. [...] > > If not covered by Assertor properties, I propose we add something to > > the strawman. > >There are two separate issues here. sbp's 1.0 proposal had both a test >mode and subclasses of Assertor (MachineAssertor and PersonAssertor). >If the mode of an assertion is automatic, should it be understood that >the assertor is a machine (or conversely, if it is manual, is the >assertor a person)? Is a manual assertion made by a machine >nonsensical? Good questions. I think automatic assertions are always made by machine assertors, but I don't think the converse is always true. I think it is possible for a machine to make a manual assertion. e.g. several accessibility evaluation tools prompt the user to perform manual tests. The tool could assert the test result and that it was derived from human input. This should also increase the confidence of the assertion. >I suspect that EARL needs only one of the two ways of indicating how a >test was performed; subclassing Assertor seems intuitively simpler, >but explicitly indicating the mode would also work adequately. It seems to me that it would make more sense to have a property of the assertion rather than subclass the assertor. I think the more interesting information is how the testResult is derived rather than knowing more about the assertor. I realize that more information about the assertor generally increases the trust that people have in the assertion, but I think in this case the test mode is a property of the assertion rather than the assertor. >The second issue concerns the meaning of the heuristic mode. I've not >been able to find a concrete definition for this, so it is difficult >to say whether or not it will be covered by Assertor properties. If I remember correctly: * automatic - the test is cut and dry. e.g. either the img element has an alt attribute or it does not * heuristic - we can check for patterns but this might not be exhaustive or it might generate a false positive. e.g. the alt-text of this image seems to be ok since it does not contain the following suspicious phrases: ".gif", "insert alt-text here", "KB", "image", etc. * manual - there are no checks that the machine performs, although it can give the user some idea of where to look for potential problems. e.g. the script element is present. I haven't seen heuristic used too much and it might not be worth keeping. The lines are fairly blurry. We could define automatic and manual as follows: automatic: the test was run without human intervention. manual: the test required human intervention. Thoughts? > > 5. TestCriteria, suite, level, excludes (from 1.0 test) > > > While these describe pieces of the testCase, at the face to face we > > agreed that we need some way to group tests. Suite, level, and > > excludes were created to do this. I propose we add something to the > > strawman. Ideas? > >I think that EARL should distinguish between groups of testcases and >groups of the criteria that the testcases test. agreed. >The line between a testcase and a criterion is also a bit blurry - are >WCAG checkpoints considered to be testcases or criteria? In WCAG 2.0 we have 3 levels: 1. Guidelines 2. Checkpoints 3. Success criteria a success criterion is the child of one checkpoint and the great-grandchild of one guideline. a checkpoint is the child of one guideline and at least one child (success criterion). a guideline has siblings (other guidelines) and more than one child (checkpoint). the root is the document "WCAG 2.0" Although, people won't be claiming conformance to Guidelines, they will only be testing against checkpoints and success criteria. Thus, I think your proposed schema would work for WCAG 2.0 conformance testing. (I tried it out below...really quick first pass...) CSS Test Suite seems to group more than one test under one title. e.g. 2.1 "anchor" [1] seems to contain 5 tests. Or are these considered one test? A {color: green;} A:link {color: purple;} A:visited {color: lime;} A:active {color: maroon;} #one {color: #006600;} [1] http://www.w3.org/Style/CSS/Test/CSS1/current/sec21.htm Using the "throwaway strawman" from [2] http://lists.w3.org/Archives/Public/w3c-wai-er-ig/2002Aug/0009.html We could come up with the following. I'm not sure how I feel about this. I didn't need to use Criterion, CriterionGroup, containsCriterion, or tests. I really like the idea of tests, tho... png of the graph is at: http://lists.w3.org/Archives/Public/www-archive/2002Sep/0110.html <rdf:RDF xmlns="http://www.w3.org/2001/03/earl/0.95#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <Assertor> <name>Bob B. Bobbington</name> <email rdf:resource="mailto:bob@example.org"/> <asserts> <Assertion> <subject rdf:resource="http://www.w3.org"/> <result rdf:resource="http://www.w3.org/2001/03/earl/0.95#passes"/> <testcase rdf:resource="http://example.org/#WCAG20-Checkpoint-1"/> </Assertion> </asserts> </Assertor> <TestGroup rdf:about="http://example.org/#WCAG20-Checkpoint-1"> <containsTestCase rdf:resource="http://example.org/#success-criterion-1"/> <containsTestCase rdf:resource="http://example.org/#success-criterion-2"/> </TestGroup> </rdf:RDF> >Excluded from this are, well, exclusions. I couldn't find any clear >indication of how these are expected to be used - examples would be >welcome. Yes, I'm not sure how to use exclusions either. I think we can drop it. > > 6. os, version (from 1.0 test) > > > Is this covered by platform. It seems to me that these are > > necessary to uniquely identify user agent test subjects. If not > > covered by platform, I propose we add something to the strawman. > >In the strawman (and in sbp's proposal), platform describes aspects of >the environment in which the assertor performed the tests, rather than >aspects of the subject of the test. In sbp's proposal, os is a >refinement of platform, and so also describes the test environment, >not the test subject. > >In the strawman, user agent test subjects are identified by a URI, >with a human-readable label being provided using rdfs:label. If it is >necessary to explicitly state the OS and version of the UA >(rather than implicitly coding this in the URI used to identify the >UA), the schema can easily be modified to include os and version >properties. If each version and os release of a UA is identifiable by a unique URI then that should use a URI. What is the URI for Opera v5 for Mac OS X? is it http://www.opera.com/download/get.pl?platform=mac&force=5.0 ? That seems to be the only thing I could find on their site. This is similar to identifying the reprof of web content with the normalization algorithm used and result. Should we include earl properties to uniquely identify test subjects (i.e. normalization algorithm, hash algorithm, hash values for web content and os, version for tools)? > > 8. date (from 1.0 test) snip, snip >For this reason, I believe that the right course to take for EARL 1.0 >on this issue would be to use the date property from the DC vocabulary >in preference to creating a date property for use with EARL alone. Sounds good to me. --wendy -- wendy a chisholm world wide web consortium web accessibility initiative seattle, wa usa /--
Received on Tuesday, 10 September 2002 18:18:13 UTC