Re: What was the test, Re: EARL Scenarios

Hi,

<andrew_kirkpatrick@wgbh.org> wrote:

> I couldn't know less about OWL, but there would potentially need to be  
> more complex referencing than 1:1 declarations like 'Same As'.  Some  
> tools do one test that others do in 5 separate tests, while in other  
> cases a tool  has a test that checks for situations 'a', 'b', and 'c'  
> and another tool has 2 tests, one that checks for 'b' and 'd' and  
> another that checks for 'a', 'c', and 'e'.

This is the case for WCAG 1.0 because the checkpoints were the most basic layer which the guidelines provided. It is apparent that this is not fine-grained enough to avoid multiple interpretations and ambiguity.

There are two strategies here which could resolve this problem, in ERT we need to focus on both and provide a basis for better quality tools:

1. Comprehensive Test Suites
 WCAG 2.0 is generating tons of test suites so that there is exactly one decision tree to determine conformance to a checkpoint (a.k.a. Success Criteria). this means, all tools will be executing tests 'a', 'b', and 'c' in order to determine checkpoint X. How well a tool implements these atomic tests will decide their precision.

2. Semantic Mapping of Tests
 OWL as well as other semantic Web languages are capable of describing more complex relationships than 1:1 mappings. A tool developer could publish their own map (i.e. decision tree) for how the tool derives conclusions about a checkpoint. The EARL assertion shows the result of the combined sequence of tests (=checkpoint).

Here an example:

Checkpoint: "1.1 provide text descriptions for non-text content"
Tests:
  'a' - each IMG must have ALT attribute
  'b' - ALT attribute must be empty for decoration images
  'c' - ALT attributes should make sense

Strategy 1:
- Tests 'a', 'b', and 'c' are defined by WCAG 2.0. They must be complete and atomic (for example, test 'c' might require further sub-tests).
- The decision tree is provided by WCAG 2.0 (if test 'b' successful, then don't execute test 'c' because it is not applicable etc)
- Tool developers "only" need to implement these tests into a programming language (for example test 'b' will be a challenge)

Strategy 2:
- Tests 'a', 'b', and 'c' are defined by the evaluation tool developer, other tool developer do not know them or use other ones
- The tool developer publishes a mapping (ideally in some form of RDF) that the success to tests 'a', 'b', and 'c' constitute the result for checkpoint 1.1
- Other tool developers only want to know if checkpoint 1.1 failed or passed. Still, they can process the mapping to find the specific error/success message of the test which failed/passed.

IMO both these strategies have pros and cons which is why I think we need to focus on both in ERT. We need to help WCAG 2.0 develop the test suites we need to build evaluation tools, but we also need to look into semantic Web technologies to allow mappings and exchange of test descriptions between the tools (not only the exchange of results).

Huch, what a long mail! Thanks for reading so far down ;)

Cheers,
  Shadi

Received on Thursday, 24 February 2005 16:36:58 UTC