Re: Test Framework for WCAG 2.0

Patrick,
Your response is very helpful.  I'll take this back to the WCAG WG to 
discuss. I expect we'll have more questions in the near future.
Thank you,
--wendy

At 01:00 AM 8/11/2003, Patrick Curran wrote:
>Wendy A Chisholm wrote:
>>Hello,
>>
>>The WCAG WG would like to begin implementation testing of WCAG 2.0 this 
>>fall.  I've read QA Framework: Test Guidelines and have a few questions 
>>about Guidelines 3 and 4.  What should a Test Framework look like? For 
>>WCAG is a test management system an evaluation tool?  The SVG Conformance 
>>Test Suite -- Test Builder's Manual [2] is helpful, but since WCAG is a 
>>set of guidelines to apply to Web content rather than a language 
>>specification it is not clear to me what our test suite will look like.
>>
>>I've drafted some initial thoughts in "DRAFT WCAG 2.0 Implementation 
>>Testing Framework" [3]
>>
>>Thoughts?
>>
>>Thanks,
>>--wendy
>>
>>[1] 
>><http://www.w3.org/QA/WG/2003/02/OpsET-charter-20030217>http://www.w3.org/QA/WG/2003/02/OpsET-charter-20030217 
>>
>>[2] http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm
>>[3] 
>><http://www.w3.org/WAI/GL/implementation-testing/Overview.html>http://www.w3.org/WAI/GL/implementation-testing/Overview.html 
>>
>
>Thanks for your questions, Wendy. Since I'm currently working on the next 
>draft of our TestGL document I'll try to respond. I apologize for the 
>length of this reply, and for the fact that I'm probably raising as many 
>questions as I'm answering.
>
>If I'm interpreting your message correctly, you are asking for 
>clarification of the terms "test management system" and "test framework", 
>particularly in the context of WCAG, since it seems somewhat difficult to 
>apply traditional testing techniques and processes to guidelines such as 
>yours.Your implementation-testing overview document also seems to raise an 
>additional question: what is implementation testing, and how should it be 
>carried out?
>
>I'll tackle these issues in reverse order: from the general to the specific:
>
>
>1) Implementation testing
>
>Implementation testing generally refers to the process of testing 
>implementations of technology specifications. This process serves the dual 
>purpose of verifying that the specification is implementable in practice, 
>and that implementations conform to the specification. This process helps 
>to improve the quality and interoperability of
>implementations.
>
>Your Overview document seems to place considerable emphasis on the 
>feedback the WCAG WG will receive during implementation testing, and on 
>the opportunities that this feedback will provide for you to improve your 
>specification (the Guidelines). This is an important process, and one that 
>we should probably explore in one of our GL documents (OpsGL?), but it 
>does seem to me that this is distinct from (though obvoiusly related to) 
>implementation testing.
>
>Rather than (or in addition to) focusing on testing implementations of 
>your specification you seem to be concerned with testing the specification 
>itself (by watching and learning from the implementation process). The 
>testing strategy you outline (the discussion of variables, for example) 
>seems to confirm this interpretation. Moreover, in the 'Questions and 
>other thoughts' and 'Implementation Framework' sections of your document 
>you seem to go beyond even the process of gathering
>feedback, and start to discuss the encouragement and promotion of 
>implementation testing itself.
>
>All of this is admirable, but it might be easier to formulate a strategy 
>if you made explicit the multiple processes and goals that seem to be 
>inherent in your document. I see at least three:
>
>a) The encouragement and support of implementation testing.
>b) The process of testing your Guidelines by encouraging their adoption 
>and by soliciting and incorporating feedback.
>c) The creation and organization of conformance test materials.
>
>The first of these items is addressed, though without much detail, in 
>Guideline 6 of TestGL: 'Plan for conformance testing'. The second - how 
>the test materials are themselves tested - does not seem to be addressed 
>in any of our documents. I'll work on adding something to TestGL. As 
>you've noted, the third is directly addressed in TestGL, albeit once again 
>without much detail.
>
>
>2) Testing techniques for Guidelines
>
>In your message you say "since WCAG is a set of guidelines to apply to Web 
>content rather than a language specification it is not clear to me what 
>our test suite will look like".
>
>While it's true that tests developed to verify conformance with guidelines 
>such as yours (and ours, in our QAWG documents) will differ from those 
>developed to verify conformance to a language specification, there are 
>nevertheless many similarities.
>
>The principal difference lies in the extent to which the test can be 
>'automated' (that is, scripted or programmed for machine-execution). While 
>it's true that some of your tests (for example, some tests for Guideline 
>1.1 'Provide a text equivalent for every non-text element') can probably 
>be programmed, most of them (for
>example, tests for Guidline 2.1: 'Ensure that all information conveyed 
>with color is also available without color') will of necessity be 
>'manual', requiring a human to follow written instructions, and to use 
>subjective judgement as to whether the results are as expected.
>
>In either case it's important to clearly define what is to be tested 
>(which of your guidelines or checkpoints), the context within which it is 
>to be tested (for example whether the content was static or was 
>programatically generated), the steps that must be taken to execute the 
>test, and the expected results.
>
>Once a test is defined in this manner, it becomes simply an implementation 
>detail whether the setup, the 'execution instructions', and the comparison 
>of actual to expected results are encoded in a computer language for 
>machine execution or in written English for human execution.
>
>The UAAG and Voice Browser WGs have experience in developing (and 
>managing, to jump ahead to the next topic) 'interactive' and 'subjective' 
>tests. If you haven't already done so, you might want to consult with them.
>
>
>3 Test Management System and Test Framework
>
>As you've noticed, our definitions of these concepts need work. Thanks for 
>encouraging us to clarify them.
>
>A Test Management System is a well-defined system for organizing and 
>classifying tests and associated metadata, such as who submitted the test, 
>the assertion that it tests, whether it's associated with optional 
>functionality or with a particular conformance level, etc. All this really 
>means is that WGs should keep track of the tests that they develop or 
>collect, and should be able to identify them, classify them, and select 
>from them as necessary.
>
>As with the tests themselves, it's more important that this system be 
>specified and documented than that it be automated.
>
>By Test Framework we mean a well-defined process for executing tests. A 
>test framework should ensure that two different testers testing the same 
>implementation should execute the same tests in the same manner and should 
>obtain identical results.
>
>If the installation, configuration, and execution of the test suite is 
>automated then these goals are more easily achieved, but if this is 
>impractical then careful specification and documentation will be sufficient.
>
>
>Thanks again for your questions, and for helping us to clarify our own 
>documents. I hope this response is at least somewhat helpful to you. 
>Please feel free to follow up - this is an interesting discussion and 
>there's a lot to be done.
>
>Patrick Curran: Sun Microsystems representative to W3C QAWG
>

-- 
wendy a chisholm
world wide web consortium
web accessibility initiative
http://www.w3.org/WAI/
/--

Received on Monday, 11 August 2003 17:58:52 UTC