Test Cases, TPDL, and Formalisms in EARL

[Apologies for the cross post, please trim as appropriate.]

Recent discussion in the ERT WG and QA activity has focussed upon the
formation of a Test Point Definition Language (TPDL), its position in the
process flow of evaluation and repair (rationale), and how best if at all
to proceed with it, if at all. This note is an attempt to summarize recent
discussion, to point out how crucial TPDL is to the future of work on EARL,
and to get some kind of rough consensus on how to move forward quickly.

As many of you will know, the ERT Working Group members (under the aegis of
the WAI, and perhaps one day QA) have been conducting work upon the
Evaluation And Repair Language (EARL [1]). This is a generic framework for
expressing evaluation information - with a lean towards rating Web content,
tools, and user agents - w.r.t technical guidelines and other criteria. As
such, it does not constitute a test point definition language of itself,
but it does have to link to test points (we call them "test cases" in EARL)
in order to make the evaluation.

For example, Chris Ridpath of the University of Toronto has been working on
a tool called "ATR" [2], which enables a human operator to rank an
accessibility tool against how well it handles a series of
accessibility-related test cases. We need to specifically point to the test
cases (all of which have URI-views) in order to rank whether the tool
passes or fails them. Note how the test case details are arranged: rather
than linking directly to the test case, we use an ID for the test case, and
then point from the ID to the test case file itself. This is because EARL
allows one to not only link to test cases, but to also come up with a wide
range of criteria, which may have a range of data associated with them:
purpose, expected result, label, whatever.

In EARL, we separated out single point criteria (e.g. a WCAG checkpoint)
from single point test cases (e.g. a DOM test case) because they did not
appear to be the same thing: passing a guideline is different from passing
a test case in that a test case is one step on from a checkpoint. The test
case may for example be a sample piece of technology that may or may not
invoke the checkpoint, or it may be a detailed model of a mechanism which
may or may not embody some specification point. Or, perhaps, both.
Separating specification points and test points may be an incorrect way of
modelling it, but a practical way, for it gets us thinking about the levels
of metadata attached to each.

In the case of the DOM TS, it is apparently a fine line between the two,
but that's irrelevant at the moment. From the draft DOM TS language DTD
[3], it is evident that there is a slight overlap between the test case
class in the EARL schema, and some of the apparent properties in the DTD
(it's difficult to tell the semantics behind a syntactic schemata). For
example, both specify a "purpose", a "description", and an "id". However,
because the DOM TS is obviously a very specialized thing, it goes on to
enumerate a range of DOM specific predicate relationships: node method,
first child, local name.

As Al Gilman pointed out [4], the optimum amount of elaboration for a test
case is when the "view is expanded just enough to capture all information
necessary to evaluate the test-requirement assertions". Further, this means
that "some details behind the interface are elided". I do not know whether
the DOM TS is being overly verbose (I doubt it), and I am not sure whether
it, or ATR, or EARL would benefit from work on a suitable abstract test
point definition framework based on (as Al put it) "state-trace" patterns.
Al's message had a third aspect to it, which was to underline the parallels
between specification and test points, which is not something that I am
necessarily enamoured with, but nevertheless is a noteworthy comment, and
required reading.

This is something that we all need to come together and discuss, and hence
the cross-post. It affects all three groups do a great degree because work
on EARL will be havily affected by the outcome of any TPDL work, and it may
well turn out that the DOM TS could heavily benefit by augmenting its work
to bring it inline with any findings. QA, as the new arbiter of the W3C's
dog food program, need to ensure that we do what's best: work on a TPDL
language would ideally (IMHO) be carried out there.

[Skippable rant about the overall process flow.]

At the recent ERT WG F2F meeting [5], we discussed the process flow of
evaluations (which we termed as the "end-to-end process"). Here is a
summary of a graph made by Dave Pawson:-

1) Requirements capture => 2) Test specification => 3) Test => 4) Test
result => 5) Collate results => 6) Analyze results (query) => 7) Present
results

Unfortunately, for it makes the model more difficult to grasp, there are a
number of other threads running along with this, that of the tools
designers, and of the instance data creators, etc.

Anyway, we kinda decided that EARL fits into step 4 of the above, and that
the TPDL will fit into step 2. Clearly, the test results must contain
(perhaps as a link by reference) the test specification material. It would
make much more sense to me if this proces flow were as automated (and hence
machine readable) as possible, because otherwise the chain breaks down.
That is not to say that human intervention does not play its part, because
the human is often the one running the test in the first place.

As an example of how this would save data, and increase interoperability,
at the moment in the ATR output, we have the test case as an ID, and then
link to the test case data. If the test case data were itself standardized,
we could simply point to that. Unfortunately, if this is so, the work on
EARL will depend greatly on the outcome of TPDL work, and so it is
imperative that this be sorted out as soon as possible.

Cheers,

[1] http://www.w3.org/2001/03/earl/
[2] http://www.aprompt.ca/ATR/
[3]
http://lists.w3.org/Archives/Public/www-dom-ts/2001May/att-0067/01-methods.
dtd
[4] http://lists.w3.org/Archives/Public/www-qa/2001Oct/0036
[5] http://www.w3.org/WAI/ER/2001/10/f2f-notes

--
Kindest Regards,
Sean B. Palmer
@prefix : <http://webns.net/roughterms/> .
:Sean :hasHomepage <http://purl.org/net/sbp/> .

Received on Monday, 15 October 2001 20:55:48 UTC