Wendy A Chisholm wrote:

Hello,

The WCAG WG would like to begin implementation testing of WCAG 2.0 this fall. I've read QA Framework: Test Guidelines and have a few questions about Guidelines 3 and 4. What should a Test Framework look like? For WCAG is a test management system an evaluation tool? The SVG Conformance Test Suite -- Test Builder's Manual [2] is helpful, but since WCAG is a set of guidelines to apply to Web content rather than a language specification it is not clear to me what our test suite will look like.

I've drafted some initial thoughts in "DRAFT WCAG 2.0 Implementation Testing Framework" [3]

Thoughts?

Thanks,
--wendy

[1] http://www.w3.org/QA/WG/2003/02/OpsET-charter-20030217
[2] http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm
[3] http://www.w3.org/WAI/GL/implementation-testing/Overview.html

Thanks for your questions, Wendy. Since I'm currently working on the next draft of our TestGL document I'll try to respond. I apologize for the length of this reply, and for the fact that I'm probably raising as many questions as I'm answering.

If I'm interpreting your message correctly, you are asking for clarification of the terms "test management system" and "test framework", particularly in the context of WCAG, since it seems somewhat difficult to apply traditional testing techniques and processes to guidelines such as yours.Your implementation-testing overview document also seems to raise an additional question: what is implementation testing, and how should it be carried out?

I'll tackle these issues in reverse order: from the general to the specific:

1) Implementation testing

Implementation testing generally refers to the process of testing implementations of technology specifications. This process serves the dual purpose of verifying that the specification is implementable in practice, and that implementations conform to the specification. This process helps to improve the quality and interoperability of
implementations.

Your Overview document seems to place considerable emphasis on the feedback the WCAG WG will receive during implementation testing, and on the opportunities that this feedback will provide for you to improve your specification (the Guidelines). This is an important process, and one that we should probably explore in one of our GL documents (OpsGL?), but it does seem to me that this is distinct from (though obvoiusly related to) implementation testing.

Rather than (or in addition to) focusing on testing implementations of your specification you seem to be concerned with testing the specification itself (by watching and learning from the implementation process). The testing strategy you outline (the discussion of variables, for example) seems to confirm this interpretation. Moreover, in the 'Questions and other thoughts' and 'Implementation Framework' sections of your document you seem to go beyond even the process of gathering
feedback, and start to discuss the encouragement and promotion of implementation testing itself.

All of this is admirable, but it might be easier to formulate a strategy if you made explicit the multiple processes and goals that seem to be inherent in your document. I see at least three:

a) The encouragement and support of implementation testing.
b) The process of testing your Guidelines by encouraging their adoption and by soliciting and incorporating feedback.
c) The creation and organization of conformance test materials.

The first of these items is addressed, though without much detail, in Guideline 6 of TestGL: 'Plan for conformance testing'. The second - how the test materials are themselves tested - does not seem to be addressed in any of our documents. I'll work on adding something to TestGL. As you've noted, the third is directly addressed in TestGL, albeit once again without much detail.

2) Testing techniques for Guidelines

In your message you say "since WCAG is a set of guidelines to apply to Web content rather than a language specification it is not clear to me what our test suite will look like".

While it's true that tests developed to verify conformance with guidelines such as yours (and ours, in our QAWG documents) will differ from those developed to verify conformance to a language specification, there are nevertheless many similarities.

The principal difference lies in the extent to which the test can be 'automated' (that is, scripted or programmed for machine-execution). While it's true that some of your tests (for example, some tests for Guideline 1.1 'Provide a text equivalent for every non-text element') can probably be programmed, most of them (for
example, tests for Guidline 2.1: 'Ensure that all information conveyed with color is also available without color') will of necessity be 'manual', requiring a human to follow written instructions, and to use subjective judgement as to whether the results are as expected.

In either case it's important to clearly define what is to be tested (which of your guidelines or checkpoints), the context within which it is to be tested (for example whether the content was static or was programatically generated), the steps that must be taken to execute the test, and the expected results.

Once a test is defined in this manner, it becomes simply an implementation detail whether the setup, the 'execution instructions', and the comparison of actual to expected results are encoded in a computer language for machine execution or in written English for human execution.

The UAAG and Voice Browser WGs have experience in developing (and managing, to jump ahead to the next topic) 'interactive' and 'subjective' tests. If you haven't already done so, you might want to consult with them.

3 Test Management System and Test Framework

As you've noticed, our definitions of these concepts need work. Thanks for encouraging us to clarify them.

A Test Management System is a well-defined system for organizing and classifying tests and associated metadata, such as who submitted the test, the assertion that it tests, whether it's associated with optional functionality or with a particular conformance level, etc. All this really means is that WGs should keep track of the tests that they develop or collect, and should be able to identify them, classify them, and select from them as necessary.

As with the tests themselves, it's more important that this system be specified and documented than that it be automated.

By Test Framework we mean a well-defined process for executing tests. A test framework should ensure that two different testers testing the same implementation should execute the same tests in the same manner and should obtain identical results.

If the installation, configuration, and execution of the test suite is automated then these goals are more easily achieved, but if this is impractical then careful specification and documentation will be sufficient.

Thanks again for your questions, and for helping us to clarify our own documents. I hope this response is at least somewhat helpful to you. Please feel free to follow up - this is an interesting discussion and there's a lot to be done.

Patrick Curran: Sun Microsystems representative to W3C QAWG