Accessibility Testing Technical Documentation (Draft) - 18 Mar 03

This draft document gives principles and associated techniques concerned with technical aspects of accessibility testing. No significance is implied in the ordering. The intent of this document is to stimulate discussion, and comments are welcome at any time. Please send comments to Tim Boland at: boland@nist.gov. A distinction is made between principles applicable to a test suite as a whole and principles associated with each test case in a test suite. This document is not a testing process document, but is meant to complement such a document when it exists.

Definition: Test Suite consists of a set of test cases, test documentation, and test software (e.g., harness, software to configure the test cases to a local environment, run the tests and capture the results)

    Test Suite Principles

    1. A test suite should have a logical and coherent structure
      • hierarchical, linear
      • functional,spec-based
      • introduction, table of contents, acknowledgements, test suite errata, related resources
      • glossary of terms
    2. Multiple means of navigation through a test suite should be possible
      • navigate test cases hierarchically, linearly, or other type of traversal
      • allow for cross section of property/unit combinations in testing
      • ability to generate only relevant test(s)
    3. It is better to have a simpler test suite earlier than to wait for a more comprehensive test suite and have nothing in the meantime
      • relation to spec evolution
    4. A test suite should contain at least one atomic test for each guideline/feature/checkpoint of a technology
    5. Different kinds of test cases should be included in a test suite
      • atomic
      • basic
      • composite
      • edge test cases
      • exception test cases
    6. Documentation should be included with a test suite
    7. The test suite should be user-friendly
      • ability to configure test suite for allowable adaptations (e.g., to create a language-specific implemenation of an abstract binding)
      • ability to enter input concerning the test suite
      • ability to supply data about tester

    Test Case Principles

    1. Every test case should have a logical and coherent structure
      • self-explanatory test purpose
      • links to relevant part(s) of specs
      • binary expected result (pass or fail)
      • obvious and intuitive test case results
    2. Documentation pertinent to each test case should be provided along with the test case itself
      • required supporting files
      • unambiguous and meaningful naming of test case
    3. Results of executing test cases should be recorded
      • archival and retrieval capability
      • machine-produced results vs. human-reportable results
    4. Presentation of results of test cases should be accessible
      • audio
      • visual
      • tactile
    5. A test case should allow for user interaction
      • comments on aspects of the test case by user
      • prompting of user for more information if needed
    6. Test cases should be easy to author
      • use of templates
      • identification of testable requirements
      • reuse of existing test cases where appropriate
    7. All test cases should be valid
      • pass HTML,CSS,XML validator, for example
    8. Test cases should be short and as simple as possible
      • minimize extraneous markup and code
      • text in place of images to save screen space, if appropriate
    9. In a test case it is important to test the technology being tested, not other technologies
      • explicit identification of technology/elements in use
    10. Test cases should use UTF-8 encoding

    Acknowledgements

    This draft document references the following:
    1. SVG Conformance Test Suite -Test Builder's Manual
    2. CSS Test Suite Documentation
    3. Requirements for UAAG 1.0 Test Suite