What conformance system should we use?

Below is a summary of some options that have been
discussed by the WG. The Group must decide what system 
it feels is the most appropriate: How will the document 
be used to measure the accessibility of a User Agent? 

NOTE: Unfortunately, I have no experience with the legal
contexts in which conformance might be tested, otherwise I
would propose the best solution right up front!

NOTE: In all of the scenarios below, prioritizing the
checkpoints is a good thing to let developers know which
ones are the most important. (Whether or not there should be
three levels is another story.)

FINAL NOTE: These aren't the only options, but I hope they
capture some of the issues we've been dealing with. Ideas
from each option may be mixed and matched as well.

Option 0) The document says nothing at all about 
   conformance. We let others decide how and when the
   document will be used to measure accessibility.

Option 1) The document proposes a "test suite" that
   should be used when determining conformance.
   We let others decide how and when to interpret the
   test suite.

   The test suite would amount to a checklist where, 
   for each checkpoint, a UA developer could check:

         a. Satisfied Natively

         b. Satisfied through Assistive Technology

            - For this option, we can specify whether
              this means:
                   i) The UA simply makes information
                      available through a standard
                      interface, or

                  ii) The UA developers can show that
                      they make the information available
                      <em>and</em> that there are products
                      on the market that may be acquired at
                      reasonable cost that make use of the
                      information and satisfy the
                      checkpoint.

         c. Not Applicable

         d. Not Satisfied

   Note that in as many cases as possible, a developer
   should check off both "a" and "b". Options "c" and "d"
   are mutually exclusive from "a" and "b". 

   One result of a test suite might be that conformance
   becomes a comparison game: one UA is "more conforming"
   than another because it's test suite results are more
   satisfactory. Will this produce a race to the bottom
   (all UAs are not very accessible but comparable) 
   or a race to the top (market and journalistic pressures 
   push all UAs towards high conformance since test suite
   results may be published)? 

Option 2) We define conformance in an open-ended, 
   common-sense manner, as in:

    2.a) To conform, UAs must implement all Priority 1
         checkpoints. However, they are not required to
         implement checkpoints that are not applicable (e.g.,
         because the UA does not support the indicated language 
         feature, user interface feature, content type,
         etc.).

   This could be modified, as in:

   2.b)  To conform, UAs must implement all Priority 1
         checkpoints. They are not required to
         implement checkpoints that are not applicable, but
         in those cases MUST make information available
         through standard interfaces and be able to demonstrate
         that, in conjunction with readily available products,
         the checkpoint is satisfied. 

         (Recall that UAs SHOULD make information available 
         through standard interfaces in general.) 
   
   There could even be two levels of conformance:

     i) Conforms natively by implementing all Priority 1 checkpoints.
    ii) Conforms by satisfying some Priority 1 checkpoints natively
        and others in conjunction with assistive technology.

   Note that it may be necessary to "help" common sense by
   stating precisely what functionality is expected from
   the UA in the text of some checkpoints, as in  
   "For those UAs that render video natively..."
   While it may not be necessary to make such statements generally
   (e.g., we don't have to say "For those UAs that render
   fonts natively..."), there may be some instances
   where we can avoid confusion or add precision. 

   If, however, the text of every checkpoint ends up containing
   such qualifications, we should consider Option 4 below.

Option 3) We define a hybrid system. For instance, 
   suppose we had a subset of Priority 1 checkpoints 
   (for fun, call them "Priority AAA") that were really important 
   for all User Agents and generic enough to be satisfied 
   by all User Agents. We might express conformance as follows:

       a) All UAs must satisfy all the Priority AAA
          checkpoints natively.
       
       b.i) UAs must also satisfy the applicable Priority 1
          checkpoints (where applicable is defined as in 
          Option 2).
   
   Variations on part "b" are also imaginable, including:

       b.ii) UAs must also satisfy the applicable Priority 1 
             checkpoints natively or be able to demonstrate
             that, in conjunction with readily available 
             products, the checkpoints are satisfied. 

Option 4) We specify well-defined subsets of Priority 1 
   checkpoints so that conformance is defined as
   satisfaction of one or more subsets. 

   What criteria should the WG use to establish subsets?
   How many should there be? Note that if the WG defines a
   finite number of subsets against which conformance 
   may be measured, this reduces the flexibility of the 
   document (although it simultaneously makes it more precise).

   (See, for example, Al Gilman's proposal to use User
   Interface Profiles as a means of establishing subsets
   [1].) 
 
   [1]
http://lists.w3.org/Archives/Public/w3c-wai-ua/1998OctDec/0344.html

   Although the checkpoints are being written
   for developers, conformance is likely to be measured in 
   terms of whether it satisfies a user's needs: 
   does a product (or products) make the Web 
   accessible/inaccessible to people with a certain 
   disability? This has a direct impact on the conformance
   system we define. Just as Priorities
   reflect the importance of a checkpoint to the user,
   conformance should be measured in terms of how well
   accessibility is achieved by a product 
   <em>with respect to the checkpoints in this
   document</em>. (Note that conformance to this
   document does <em>not</em> simply mean that a UA is 
   accessible, but that the UA satisfies this document's
   definition of conformance.)

   Another question: are subsets necessary at all? Will they
   be useful? Will they be useful to organizations 
   attempting to determine conformance? 
   While creating subsets is one response to the comment
   "There are too many Priority 1's to satisfy," it
   is not the only possible response and may not be useful 
   in practice.

 - Ian

-- 
Ian Jacobs (jacobs@w3.org) 
Tel/Fax: (212) 684-1814 
http://www.w3.org/People/Jacobs

Received on Wednesday, 6 January 1999 11:09:39 UTC