A Fatal Flaw and Other Problems

Date: 16 December 1998, 18:30 hrs
To: WAI-GL List
From: Eric Hansen
Re: A Fatal Flaw and Other Problems

PART 1: INTRODUCTION

I believe that there is a fatal flaw in the priority rating system and that 
there are other serious problems that should be corrected.

In order to make myself clear, I feel that I must take the role of a 
"devil's advocate," poking hard at the weaknesses as I see them. In making 
these criticisms, I do not presume that I could do a better job if I were 
in the shoes of the working group. In a task so large, we need the help of 
others to point out possible issues for resolution. We are still at a point 
at which many if not all these issues can be addressed with relative ease. 
Later on, it will not be so easy.

PART 2: A FATAL FLAW

Section 2.A. A Fatal Problem with the Rating System

Essentially the current priority rating system confounds two variables, 
making it impossible to properly interpret the ratings. I regard this as an 
extremely serious ("fatal") problem.

Following are explanations of the rating levels as found in the 7 December 
1998 version of the page authoring guidelines.

[PRIORITY 1] 
This guideline must be followed by an author, or one or more groups of 
users will find it impossible to access information in the document. 
Implementing this guideline is a basic requirement for some groups to be 
able to use Web documents. 

[PRIORITY 2] 
This guideline should be followed by an author, or one or more groups of 
users will find it difficult to access information in the document. 
Implementing this guideline will significantly improve access to Web 
documents. 

[PRIORITY 3] 
This guideline may be followed by an author to make it easier for one or 
more groups of users to access information in the document. Implementing 
this guideline will improve access to Web documents. 

(Source: 7 December 1998 Version of the Page Authoring Guidelines, 
http://www.w3.org/WAI/GL/WD-WAI-PAGEAUTH-19981207/)

Note that each priority has two aspects, (1) impact (2) and imperative for 
use. 

In order to clarify these two aspects, I have separated below the three 
"impact" levels from the three "imperative for use" levels.

The "impact" aspect addresses the severity of the usability problems caused 
by violation of the technique.  Below, I have separated this aspect. (I 
have also changed the document to refer to techniques instead of guidelines,
 since I believe that the editors are making that change.)


[Impact Level 1] 
Violations of this technique result in one or more groups of users finding 
it impossible to access information in the document. Implementing this 
guideline is a basic requirement for some groups to be able to use Web 
documents. 

[Impact Level 2] 
Violations of this technique result in one or more groups of users finding 
it difficult to access information in the document. Implementing this 
guideline will significantly improve access to Web documents. 

[Impact Level 3] 
Violations of this technique may result in one or more groups of users 
finding it somewhat difficult for one or more groups of users to access 
information in the document. Implementing this guideline will improve 
access to Web documents. [Note. I have made a minor editing change to this 
priority in order to make it more parallel to the first two. These minor 
changes are immaterial to the argument.]
(Adapted by Eric Hansen from the 7 Dec 1998 page authoring guidelines.)

On the other hand, the "imperative for use" aspect indicates how important 
it is for the page author to adhere to the technique. Below, I have 
separated out the "imperative for use" portion. (I have also changed the 
document to refer to techniques instead of guideline, since I understand 
priorities are being removed from guidelines.)

[Imperative Level 1] 
This technique must be followed by an author.
 
[Imperative Level 2] 
This technique should be followed by an author.
 
[Imperative Level 3] 
This technique may be followed by an author.
(Adapted by Eric Hansen from the 7 Dec 1998 page authoring guidelines. 
Note: I would actually prefer to phrase these differently, but those 
changes are immaterial to the argument.)

I submit that these two aspects -- impact and imperative for use -- should 
be separated. And the fact that they are not separated in the 7 December 
guidelines document is a very serious problem.

Because each priority level inextricably links both impact and imperative 
for use, the two aspects cannot vary independently of each other. It makes 
no sense to confound the operation of these two variables.

This leads to disastrous results. For example, when a technique is given a 
low priority, one does not know whether (a) it is because violation of the 
technique was not serious, (b) because the working group believes that is 
not imperative (important) that the technique be used, or (c) both (a) and  
(b). 

This tight coupling of the "impact" and "imperative for use" aspects is a 
serious, indeed, a fatal source of bias. 

Section 2.B. A Possible Example

Let us consider a situation that might arise. Suppose that an extremely 
serious accessibility problem is identified but that the problem is only 
poorly addressed by page markup and a full solution is extremely expensive. 
Suppose that the high expense of making the change causes the working group 
to rate it as having a low imperative for use. The rating system requires 
that impact and imperative for use match each other, so the working group 
has essentially four choices. First, they can adjust the impact level down 
to make it match the imperative for use. Second, they can raise the 
imperative for use rating, making it match the impact level. Third, they 
can both lower the impact level _and_ raise the imperative for use, thereby 
making them match each other. Fourth and finally, they can simply ignore 
the accessibility problem. 

Section 2.C. An Example From the Document

I believe that this flaw is already causing significant problems.

For example, consider guideline A.8 (Ensure tables have necessary markupț) 
is listed as a priority 1 guideline but a note says "[Editor: no P1 
techniques here.]." In the A.8 section of the document there is a note 
asking "Should this be Pri 1?"

My interpretation of this situation is that the working group judged 
guideline A.8 to address a severe (high "impact") problem regarding tables, 
such that violation of the guideline would prevent use by one or more 
groups. So they gave the guideline a priority 1 rating. Yet there was no 
technique that they could heartily endorse and give a priority 1 rating. 
Why did they not declare a clear priority 1 to a technique? There are many 
possible reasons for this. Perhaps there were concerns about the 
feasibility, cost, or effectiveness of the available techniques, so they 
chose to give priority 2 or 3 ratings to the techniques. This is just 
another example show that the guidelines are using the term "priority" to 
represent an impossible amalgam of "impact" and "imperative to use." (Note. 
My basic argument is unaffected by the fact that priorities are being 
removed from the guidelines.)

PART 3: BIAS AGAINST EXPENSIVE PROBLEMS

Section 3.A. Possible Bias

The foregoing example leads to another major problem, i.e., that the 
document may be biased against addressing issues that cannot be solved 
inexpensively through page markup. This bias, if correct, is significant 
because it appears to have led to other problems that further threaten the 
validity of the document.

The guidelines appear biased toward issues that can be fixed _at a low cost 
through page markup_ but appears biased against issues that are _more 
costly or difficult to solve_, even if the seriousness of the problem might 
justify the cost. The document's current stance appears to be that if a 
problem that cannot be addressed cheaply through page markup then either it 
does not exist or it is relegated to a lower priority. Possible examples of 
such issues include language, cultural, sensitivity, legal, economic, 
security, and marketing issues. (I do not intend to say that there are no 
inexpensive techniques in these domains.) 

It may seem strange to refer to an unstated emphasis on inexpensive 
solutions as a form of bias. What would be the causes and consequence of 
such a bias?

Section 3.B. A Possible Motivator for the Bias

Why would there be a bias against expensive solutions, even when the 
solutions might be justified by the seriousness of the accessibility 
problem?

Perhaps it boils down the one thing, a concern for "marketing" of the 
document. Perhaps the working group is concerned that if Web developers 
think that following the accessibility guidelines will be expensive, they 
won't pay any attention to the guidelines. Evidence for this marketing 
concern is found in a statement in the introduction to section A: 
"_Creating pages that transform gracefully is not more costly_, but 
requires a different design approach that also makes page compatible with 
emerging mobile technologies" (emphasis added). Frankly, this sounds like 
marketing hype. The document presents no evidence that this assertion is 
true. What may be true, and could perhaps be asserted, is that "Many of the 
techniques can be implemented at little or no extra cost, especially at the 
design stage of Web development." But to state categorically that the 
relevant techniques (evidently A.1 through A.12) will not cost more seems 
untenable. If it were mere puffery, it would not be so important. But a 
statement like that further paints the guidelines document into a corner. 
In order to avoid evidence that might disconfirm the assertion that the 
techniques cost no additional money, the document might either downgrade 
the imperative for use of expensive techniques or eliminate any reference 
to the expensive accessibility problems. This does Web developers a 
disservice. By failing to warn them of the existence of the problems, they 
are thereby hindered in their ability to address those problems. 

Is it wrong to be concerned for the marketing or acceptance of the 
document? No. It is absolutely necessary. But I think that marketing of the 
document will be most successful if we tell things like they really are and 
help Web developers make wise accessibility-related decisions.

Section 3.C. Other Effects

I think that there is evidence that the bias may have affected the document 
in other ways.

Consider, for example, the issue of "language" that I have raised 
repeatedly. Technique B.3 now addresses the issue partially, but the 
technique is relegated to a priority 2 level. Is it possible that the issue 
is really a high impact issue (level 1) but that the expense involved in 
addressing it the caused it to have very low imperative for use (level 3) 
in the eyes of the working group, so a compromise was reached to give it a 
level 2 priority?

I have suggested many other issues that could be addressed in the 
guidelines (e.g., language, cultural, sensitivity, legal, economic, 
marketing, etc.), but, except for partial treatment of the language issue, 
to best of my knowledge, the other suggestions have been ignored. Is it 
possible that the working group could not see a way to address the issues 
cheaply via page markup and that, therefore, they simply chose to act as 
though the accessibility problems do not exist?

The bias against expensive, non-markup-related problems may contribute to 
another kind of bias. The focus on the inexpensive, non-markup-related 
problems places individuals who are deaf or who have language-related 
disabilities at a disadvantage relative to individuals who are blind. Most 
of the techniques benefit individual who are blind much more than they help 
individuals with learning disabilities or deafness. Is this fair? Many of 
the accessibility issues that I have raised would benefit individuals who 
are deaf or have learning disabilities yet these have been largely ignored. 
I wonder if part of the working group's apparent reticence about a 
systematic tabulation of impacts is that it will make this fact so 
apparent.

Do I have a fundamental problem about the document giving more emphasis to 
problems faced by individuals who are blind? No. As long as the scope of 
the document and the procedures for producing the ratings are documented 
and justified, I have no problem with it.

Do I have a fundamental disagreement with having the document focus on 
issues that can be addressed relatively inexpensively through page markup? 
No. However, what I do have a deep concern about is a failure to explicitly 
state this as well as failure to acknowledge (by name) several other 
important and known categories of accessibility-related issues.

PART 4: WHAT IS THE SOLUTION?

I believe that in order to address these problems, the following must be 
done. 

#1. Uncouple the "impact" ratings from "imperative for use" ratings. They 
are separate and must be allowed to vary independently (although, in my 
view, the impact ratings inform the generation of the imperative for use 
ratings). The impact rating indicates the severity of the _problem_ while 
the imperative for use rating is essentially a rating of the importance of 
implementing the technique as a _solution_. I believe that to keep the two 
ratings together would continually and fatally bias and confuse the 
interpretation of the ratings; it would also perpetuate the misconception 
that cost considerations are unimportant or irrelevant to the document.

#2. Affirm the existence of other important Web accessibility problems even 
if you don't plan to provide specific guidelines or techniques for 
addressing them.

#3. Minimize accusations of bias by telling the truth about the document's 
scope and limitations. Provide a principled rationale or justification for 
the exclusions. I don't think that this is that hard to do. It just has to 
be done.

#4. Do not deny the role of cost considerations in the document. 
(a) Do not deny that solving some accessibility issues will cost money.
(b) Do not deny that the working group considers matters of cost in 
determining the "imperative for use" ratings or even in determining which 
problems will be addressed by techniques. Financial considerations 
essential for generating "imperative for use" ratings. (The "impact" 
ratings would be much less affected by cost considerations.)

#5. Describe how the ratings are produced.

#6. Describe the qualifications of the individuals or group that produced 
the ratings. (Note: I believe that failure to do any of #1 through #5 
places an excessive burden -- probably an impossible burden -- upon the 
expertise and authority of the people making the judgements. I don't think 
that it is fair to expect anyone to shoulder that burden.) How were people 
with disabilities represented among those developing the ratings? 


=============================
Eric G. Hansen, Ph.D.
Development Scientist
Educational Testing Service
ETS 12-R
Rosedale Road
Princeton, NJ 08541
(W) 609-734-5615
(Fax) 609-734-1090
E-mail: ehansen@ets.org

Received on Wednesday, 16 December 1998 18:35:21 UTC