Draft Test Suite for ATAG (based on 25 June 04 ATAG WD) Version 1 - August 26 2004

Background

The scope of this test suite is the ATAG2.0 25 June 04 WD; the purpose of this test suite is to provide a mechanism for demonstrating and promoting conformance to the ATAG2.0 25 June 04 WD. This test suite consists of this main page, developer and tester tests, and a reporting page for results. It is possible that all of these components of the test suite, as well as the test suite itself, may be tested prior to actual operation. All tests are required, per the conformance information in (reference? to) ATAG2.0 Conformance Section. This test suite will be hosted at a WAI site in the future, and may use applicable W3C WCAG (or UAAG) test suite material as necessary.

The philosophy of this test suite entails the concept of "roles" in testing. There are two roles defined: (1) the role of one (called "developer") who is knowledgeable of the authoring tool, and provides (necessary?) data to enable/assist the tester to run the tests, and (2) the role of one or more author(s) (called "tester(s)" - no prior knowledge of tool assumed) who may take this knowledge and run the tests, as specified following, to evaluate the claims made (by the developer?) on behalf of the authoring tool. Developer and tester may be the same individual, and more than one tester can run tests on the same tool.

NOTE: These tests will be represented as templates (interactive forms) to be completed and submitted to another site. For now they are just HTML files. The test procedure (process of executing a test) is for a template (form): take the actions indicated, fill out the entries, and submit (to testreports site). Specific results of filling out the templates (forms) should be reproducible to the maximum extent possible, given the possible human interpretation of test results for certain of the tests. Reporting of results may be by machine or by human.

At this point in time this document is informative (not-normative). Multiple means of navigation are possible through this document (maybe use XML test description with XSLT?). It is possible that in the future different kinds of tests or more complex tests may be included. It is possible that this test suite may be "configurable" in the future. For information on how to use this test suite, go to ?. For ATAG conformance info (including references to the ATAG spec/techniques), go to ? For W3C QA info, go to ?. For other WAI testing resources, go to ? The reference specifications for these tests are given in this document.

Naming conventions for templates (forms) indicate name of spec (atag), developer or user, and success criterion number (where relative priority, a dash follows to give level 1, 2, or 3). The templates (forms) may be submitted (navigated) in any order; there are no "prerequisites" for use of any template. One may go directly from one template (form) to the next (or previous), or come back to this page for a different ordering (NOTE: Are these templates "approved" by the AUWG? Assuming so, the current status of each template (form) may be given as: under review, accepted, or rejected, according to specific criteria to be determined). Each template also links to a log history, containing, for example, creator of template, modification history, etc.). Are these templates "normative"? Does the AUWG want to define what PASS/FAIL mean? If you would like to suggest new templates (forms) or changes to existing templates (forms), please submit your information to ?

NOTE: Tool designer is assumed knowledgable of tool; author is one who uses tool (no prior knowledge of tool assumed, but can be the same as tool designer, and can be many authors performing tests on a single tool). Test plan may include combinations of machine-testing and human-controlled testing. "Form" below includes a public listing of pertinent test data (including test number, success criteria number, individuals involved, and results.).

NOTE: At this point only Success Criteria 1.1 and 1.2 links work (have actual templates (forms)); if group feels this is a good approach, I can prepare the other links.

This document is organized as follows: first comes the test report site where all test reports submitted for all templates are collected, and then comes the actual templates (in order of ATAG conformance level and success criteria within that level)

Test Report Site for All Tests

ATAG2.0 Conformance Level A

  1. Success Criteria 1.1: The authoring interface must conform to ISO16071 level 1 (ISO16071 RELATIVE PRIORITY)

    Test Templates (Forms)

  2. Success Criteria 1.2: At least one editing method must conform to ISO16071 level 1 for each element and object property editable by the tool (ISO16071 RELATIVE PRIORITY)

    Test Templates (Forms)

  3. Success Criteria 1.3:
    1. All editing views must display text equivalents for any non-text content
    2. All editing view must either respect operating system display settings (for color, contrast, size, and font) or, from within the tool, provide a means of changing color, contrast, size, and font, without affecting the content markup)

    Test Templates (Forms)

  4. Success criteria 2.1: All markup string written automatically be the tool (i.e., not authored "by hand") must conform to the applicable markup language specification

    Test Templates (Forms)

  5. Success Criteria 2.2:
    1. The authoring tool must support at least one WCAG-capable format for each Web content type produced
    2. When format selection is automatic, the selected format must be WCAG-capable

    Test Templates (Forms)

  6. Success Criteria 2.3: Tools must always meet at least one of the following:
    1. generate accessible content automatically
    2. provide a method for authoring "by hand"
    3. provide the author with accessible options for every authoring task

    Test Templates (Forms)

    ------------------------

  7. Success Criteria 2.5: Unless the author explicitly instructs the authoring tool otherwise, all content generated by the tool must satisfy all of the WCAG2.0 06/02/04 WD Level 1 success criteria(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  8. Success Criteria 2.6: Any web content (e.g., templates, clip art, multimedia objects, scripts, applets, example pages, etc.) that is bundled or preferentially licensed (i.e., better terms for users of the authoring tool than for the general public) must satisfy the Level 1 WCAG2.0 (06/02/04 draft) success criteria (WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  9. Success Criteria 3.1:
    1. When the actions of the author risk creating accessibility problems (not satisfying any of the WCAG2.0 (06/02/04) Level 1 success criteria), the tool must introduce the appropriate accessible authoring practice.
    2. The intervention must occur at least once before completion of authoring (e.g., final save, publishing, etc.)
    (WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  10. Success Criteria 3.2: The tool must provide a check (automated check, semi-automated check or manual check) for detecting violations of each Level 1 success criteria of WCAG2.0 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  11. Success Criteria 3.3: The tool must provide a repair (automated repair, semi-automated repair or manual repair) for correcting violations of each Level 1 success criteria of WCAG2.0 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

--------------------------------------------------------------------------------------------------

ATAG2.0 Conformance Level AA (after exercising ATAG2.0 Conformance Level A)

  1. Success Criteria 1.1: The authoring interface must conform to ISO16071 Level 2 (ISO16071 RELATIVE PRIORITY)

    Test Templates (Forms)

  2. Success Criteria 1.2: At least one editing method must conform to ISO16071 Level 2 for each element and object property editable by the tool (ISO16071 RELATIVE PRIORITY)

    Test Templates (Forms)

  3. Success Criteria 1.4:
    1. In any element hierarchy, the author must be able, with a device-independent action, to move editing focus from any structural element to any element immediately above, immediately below, immediately below or in the same level in the hierarchy
    2. In any element hierarchy, the author must be able, with a device-independent action, to select, copy, cut and paste any element, and its content

    Test Templates (Forms)

  4. Success criteria 1.5:
    1. The authoring tool must have a search function for all editing views
    2. The author must be able to search for text within all text equivalents of any rendered non-text content
    3. The author must be able to specify whether to search content, markup, or both

    Test Templates (Forms)

  5. Success Criteria 2.4:
    1. During all transformations and conversions, all unrecognized markup and accessibility information must be preserved, unless prevented by limitations of the target format
    2. When unrecognized markup or accessibility information cannot be preserved during a conversion or transformation, the author must be notified before any change is made.

    Test Templates (Forms)

  6. Success Criteria 3.4:
    1. When the author inserts an unrecognized non-text object, the tool must not insert an automatically generated text equivalent (e.g. label generated from the file name)
    2. When the author inserts a non-text object for which the tool has a previously authored equivalent (i.e. created by the author, tool designer, pre-authored content developer, etc.), but the function of the object is not known with certainty, the tool must prompt the author to confirm insertion of the equivalent. However, where the function of the non-text object is known with certainty (e.g. "home button" on a navigation bar, etc.), the tool may automatically insert the equivalent.

    Test Templates (Forms)

  7. Success Criteria 3.7: All features that play a role in creating accessible content must be documented in the help mechanism.

    Test Templates (Forms)

  8. Success Criteria 3.8:
    1. All examples of markup code and views of the authoring interface (dialog screenshots, etc.) must satisfy all Level 1 and 2 success criteria below of WCAG2.0 (06/02/04 draft), regardless of whether the examples are intended to demonstrate accessibility authoring practices
    2. All descriptions of authoring processes must integrate the steps needed to create accessible content

    Test Templates (Forms)

  9. Success Criteria 4.1: Any mechanism that guides the author in sequencing authoring actions (e.g., design aids, wizards, templates) must integrate promting, checking, repair functions and documentation

    Test Templates (Forms)

  10. Success Criteria 4.2: When an authoring action has several markup implementations (e.g., changing the color of text with presentation markup or style sheets), those markup implementations that satisfy all of the Level 2 success criteria of WCAG2.0 (06/02/04 draft) must have equal to or higher prominence of the following scales than those markup implementations that do not meet the above WCAG2.0 requirements

    Test Templates (Forms)

  11. Success Criteria 4.3:
    1. Continuously active processes (e.g. a checker that underlines errors as they occur, a checker that runs at each save, a checker that runs every 10 minutes, etc.) that implement functions required by checkpoints 3.1, 3.2, 3.3, and 3.7 must be enabled by default
    2. If the author chooses to disable these continuously active processes, then the tool must inform the author of the consequences of their choice
    3. The accessibility prompting, checking, repair and documentation must have at least the same prominence as prompting, checking, repair and documentation for other mandatory information in the tool (e.g., prompting for file names during saves or checking for and repairing spelling or syntax errors)

    Test Templates (Forms)

  12. Success Criteria 4.X: The configurability of functions related to accessibility prompting, checking, repair, and documentation must be equivalent to that of comparable functions in terms of number of options under author control and the degree to which each option can be controlled.

    Test Templates (Forms)

    ---------------------

  13. Success Criteria 2.5: Unless the author explicitly instructs the authoring tool otherwise, all content generated by the tool must satisfy all of the WCAG2.0 06/02/04 WD Level 2 success criteria(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  14. Success Criteria 2.6: Any web content (e.g., templates, clip art, multimedia objects, scripts, applets, example pages, etc.) that is bundled or preferentially licensed (i.e., better terms for users of the authoring tool than for the general public) must satisfy the Level 2 WCAG2.0 (06/02/04 draft) success criteria (WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  15. Success Criteria 3.1:
    1. When the actions of the author risk creating accessibility problems (not satisfying any of the WCAG2.0 (06/02/04) Level 2 success criteria), the tool must introduce the appropriate accessible authoring practice.
    2. The intervention must occur at least once before completion of authoring (e.g., final save, publishing, etc.)
    (WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  16. Success Criteria 3.2: The tool must provide a check (automated check, semi-automated check or manual check) for detecting violations of each Level 2 success criteria of WCAG2.0 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  17. Success Criteria 3.3: The tool must provide a repair (automated repair, semi-automated repair or manual repair) for correcting violations of each Level 2 success criteria of WCAG2.0 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

----------------------------------------------------------------------------------------------------

ATG2.0 Conformance Level AAA (after exercising ATAG2.0 Conformance Levels A and AA)

  1. Success Criteria 1.1: The authoring interface must conform to ISO16071 level 3 (ISO16071 RELATIVE PRIORITY)

    Test Templates (Forms)

  2. Success Criteria 1.2: At least one editing method must conform to ISO16071 level 3 for each element and object property editable by the tool (ISO16071 RELATIVE PRIORITY)

    Test Templates (Forms)

  3. Success Criteria 3.5: When objects, for which alternative equivalents have been previously provided, are inserted, the tool must always offer those alternative equivalents for reuse or modification

    Test Templates (Forms)

  4. Success Criteria 3.6: The tool must provide the author with an option to view a listing of all current accessibility problems.

    Test Templates (Forms)

  5. Success Criteria 4.4: The authoring interface for accessibility prompting, checking, repair and documentation must be equivalent to the authroing interface for comparable functions in terms of the following characteristics:
    1. design (measured by design metaphors, artistic sophistication, sizes, fonts, colors)
    2. operation (measured by degree of automation, number of actions for activation)
    3. comprehensiveness (measured by breadth and depth of functionality coverage)

    Test Templates (Forms)

    ------------------

  6. Success Criteria 2.5: Unless the author explicitly instructs the authoring tool otherwise, all content generated by the tool must conform to the Level 3 success criteria of WCAG2 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  7. Success Criteria 2.6: Any web content (e.g., templates, clip art, multimedia objects, scripts, applets, example pages, etc.) that is bundled or preferentially licensed (i.e., better terms for users of the authoring tool than for the general public) must satisfy the Level 3 success criteria of WCAG2.0 (06/02/04 draft) (WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  8. Success Criteria 3.1:
    1. When the actions of the author risk creating accessibility problems according to the Level 3 success criteria of WCAG2.0 (06/02/04 draft) the tool must introduce the appropriate accessible authoring practice.
    2. The intervention must occur at least once before ocmpletion of authoring (e.g., final save, publishing, etc.)
    (WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  9. Success Criteria 3.2: The tool must provide a check (automated check, semi-automated check or manual check) for detecting violations of each Level 3 success criteria of WCAG2.0 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

  10. Success Criteria 3.3: The tool must provide a repair (automated repair, semi-automated repair or manual repair) for correcting violations of each Level 3 requirement of WCAG2.0 (06/02/04 draft)(WCAG RELATIVE PRIORITY):

    Test Templates (Forms)

    References

    1. ATAG 2.0 WD 25 Jun 2004
    2. WCAG 2.0 WD 2 Jun 2004
    3. ISO 16071: 2002(E)