- From: Gregg Vanderheiden <po@trace.wisc.edu>
- Date: Thu, 17 Dec 1998 10:58:36 -0600
- To: "'GL - WAI Guidelines WG'" <w3c-wai-gl@w3.org>
Hi Eric
RE: THE RATING SYSTEM AND BIAS
I don't quite follow your fatal flaw argument. The impact IS the determiner
of the imperative. That is, if the item is required for access, then it is
priority 1. At least that is what we were using as the rule. This rule was
worked out after much discussion early in the process.
The judgements as to what became a priority one then were made based upon
asking the question "is it possible to access the information under question
if this technique were not done?" Each judgement was either taken from
listserv discussion comments or made by the editors based on the discussion
and then put before the whole working group for discussion and confirmation.
(and then the whole ig and other external review groups and then the public)
Having two sets of ratings sounds like it would be very confusing.
Especially since the two of them would track each other.
You said that some things were rated lower because they were difficult. Do
you have an example of this? We are not aware of any.
There were some problems that existed when we had ratings on the Guidelines…
but as you noted they have been removed from the guidelines. Your comments
in previous postings on that only confirmed our tentative decision to pull
them and they are now gone. Your example was of a guideline priority
problem (now gone). Do you have any examples with techniques?
RE: ACCESS HAS NO COST
Your comment about the document saying there is no cost for accessibility is
a good one. That statement goes beyond what it was supposed to. Access
does not necessarily mean more cost - but it can and in some cases does.
We will amend that statement to say
"_Creating pages that transform gracefully is not NECESSARILY more costly_,
but requires a different design approach that also makes page compatible
with emerging mobile technologies"
You are also correct that the statement is there to help build the case for
using the guidelines. Thanks for pointing out the overstatement.
RE: TOPICS LIKE cultural, sensitivity, legal, economic, marketing, etc THAT
ARE NOT COVERED IN THE DOC
YOU WROTE:
I have suggested many other issues that could be addressed in the guidelines
(e.g., language, cultural, sensitivity, legal, economic, marketing, etc.),
but, except for partial treatment of the language issue, to best of my
knowledge, the other suggestions have been ignored. Is it possible that the
working group could not see a way to address the issues cheaply via page
markup and that, therefore, they simply chose to act as though the
accessibility problems do not exist?
RESPONSE
The reason these were not included is not because of cost issues. It is
because these are not disability access issues. A person with a disability
is not denied access to the information because of cultural or sensitivity
issues etc any more than a person without a disability. The focus of the
guidelines is on accessibility not on good design. (though we were sorely
tempted to put some good practices into the guidelines that were not access
oriented, we either resisted or ended up pulling them back out).
RE: GREATER FOCUS ON PEOPLE WHO ARE BLIND
This issue has come up many times. Help us understand here what you mean
exactly.
The document does not have (that we know of ) a greater focus on people who
are blind. It addresses every problem we could think of for every
disability type. At least that has always been the goal. There are
currently more blindness barriers than there are for say people with hearing
disabilities. And most physical disabilities are not effected by page
design as much as by the design of the User Agent. Cognitive is probably
the toughest one. We have techniques for people with no sight or no hearing
or even physical movement. But we have no solutions for providing access to
most content on the web for people who have no cognition…. Or almost no
cognition… or even moderate to severe impairment of cognition. This area
starts to move from how the information is presented to the content of the
information itself.
If you have any specific guidelines or techniques that would increase access
for people with any type of disability please post them to the list pronto.
We don't know of any that were missed (but there has been a lot of traffic).
REGARDING YOUR 6 ACTION ITEMS
Each item is below followed by a comment. The comments are labeled GV:
#1. Uncouple the "impact" ratings from "imperative for use" ratings. They
are separate and must be allowed to vary independently (although, in my
view, the impact ratings inform the generation of the imperative for use
ratings). The impact rating indicates the severity of the _problem_ while
the imperative for use rating is essentially a rating of the importance of
implementing the technique as a _solution_. I believe that to keep the two
ratings together would continually and fatally bias and confuse the
interpretation of the ratings; it would also perpetuate the misconception
that cost considerations are unimportant or irrelevant to the document.
GV: We are only rating the importance of the proposed solutions or
techniques. We are not rating the problems. As you suggest, the ratings of
the techniques are derived from the severity of the problem they address.
#2. Affirm the existence of other important Web accessibility problems even
if you don't plan to provide specific guidelines or techniques for
addressing them.
GV: Which ones are you referring to? Be happy to do this.
#3. Minimize accusations of bias by telling the truth about the document's
scope and limitations. Provide a principled rationale or justification for
the exclusions. I don't think that this is that hard to do. It just has to
be done.
GV: Made an edit to address the example you cited. Are there others?
#4. Do not deny the role of cost considerations in the document.
(a) Do not deny that solving some accessibility issues will cost money.
(b) Do not deny that the working group considers matters of cost in
determining the "imperative for use" ratings or even in determining which
problems will be addressed by techniques. Financial considerations
essential for generating "imperative for use" ratings. (The "impact"
ratings would be much less affected by cost considerations.)
GV: a) we have changed the edit to cover that
b) where do you see financial considerations being used. We are not aware
of any. They were not part of the criteria for any work group member that I
know of.
#5. Describe how the ratings are produced.
GV: We thought the definitions did that. We will relook. Check the
latest version when it is posted.
#6. Describe the qualifications of the individuals or group that produced
the ratings. (Note: I believe that failure to do any of #1 through #5
places an excessive burden -- probably an impossible burden -- upon the
expertise and authority of the people making the judgements. I don't think
that it is fair to expect anyone to shoulder that burden.) How were people
with disabilities represented among those developing the ratings?
GV: Offda. There are a lot of people in the workgroup. Or do you mean
just the editors and chairs? The rating system however was developed by the
working group after A LOT of discussion. And yes, there are people with
disabilities on the working group (which is open) though we have not asked
people to identify their disabilities.
I hope I captured all your comments. There has also been a lot of
additional discussion on the list on many of these topics that is very good
to review.
Looking forward to your thoughts.
Gregg
-- ------------------------------
Gregg C Vanderheiden Ph.D.
Professor - Human Factors
Dept of Ind. Engr. - U of Wis.
Director - Trace R & D Center
Gv@trace.wisc.edu, http://trace.wisc.edu/
FAX 608/262-8848
For a list of our listserves send "lists" to listproc@trace.wisc.edu
-----Original Message-----
From: w3c-wai-gl-request@w3.org [mailto:w3c-wai-gl-request@w3.org] On Behalf
Of eric hansen
Sent: Wednesday, December 16, 1998 5:30 PM
To: w3c-wai-gl@w3.org
Subject: A Fatal Flaw and Other Problems
Date: 16 December 1998, 18:30 hrs
To: WAI-GL List
From: Eric Hansen
Re: A Fatal Flaw and Other Problems
PART 1: INTRODUCTION
I believe that there is a fatal flaw in the priority rating system and that
there are other serious problems that should be corrected.
In order to make myself clear, I feel that I must take the role of a
"devil's advocate," poking hard at the weaknesses as I see them. In making
these criticisms, I do not presume that I could do a better job if I were
in the shoes of the working group. In a task so large, we need the help of
others to point out possible issues for resolution. We are still at a point
at which many if not all these issues can be addressed with relative ease.
Later on, it will not be so easy.
PART 2: A FATAL FLAW
Section 2.A. A Fatal Problem with the Rating System
Essentially the current priority rating system confounds two variables,
making it impossible to properly interpret the ratings. I regard this as an
extremely serious ("fatal") problem.
Following are explanations of the rating levels as found in the 7 December
1998 version of the page authoring guidelines.
[PRIORITY 1]
This guideline must be followed by an author, or one or more groups of
users will find it impossible to access information in the document.
Implementing this guideline is a basic requirement for some groups to be
able to use Web documents.
[PRIORITY 2]
This guideline should be followed by an author, or one or more groups of
users will find it difficult to access information in the document.
Implementing this guideline will significantly improve access to Web
documents.
[PRIORITY 3]
This guideline may be followed by an author to make it easier for one or
more groups of users to access information in the document. Implementing
this guideline will improve access to Web documents.
(Source: 7 December 1998 Version of the Page Authoring Guidelines,
http://www.w3.org/WAI/GL/WD-WAI-PAGEAUTH-19981207/)
Note that each priority has two aspects, (1) impact (2) and imperative for
use.
In order to clarify these two aspects, I have separated below the three
"impact" levels from the three "imperative for use" levels.
The "impact" aspect addresses the severity of the usability problems caused
by violation of the technique. Below, I have separated this aspect. (I
have also changed the document to refer to techniques instead of guidelines,
since I believe that the editors are making that change.)
[Impact Level 1]
Violations of this technique result in one or more groups of users finding
it impossible to access information in the document. Implementing this
guideline is a basic requirement for some groups to be able to use Web
documents.
[Impact Level 2]
Violations of this technique result in one or more groups of users finding
it difficult to access information in the document. Implementing this
guideline will significantly improve access to Web documents.
[Impact Level 3]
Violations of this technique may result in one or more groups of users
finding it somewhat difficult for one or more groups of users to access
information in the document. Implementing this guideline will improve
access to Web documents. [Note. I have made a minor editing change to this
priority in order to make it more parallel to the first two. These minor
changes are immaterial to the argument.]
(Adapted by Eric Hansen from the 7 Dec 1998 page authoring guidelines.)
On the other hand, the "imperative for use" aspect indicates how important
it is for the page author to adhere to the technique. Below, I have
separated out the "imperative for use" portion. (I have also changed the
document to refer to techniques instead of guideline, since I understand
priorities are being removed from guidelines.)
[Imperative Level 1]
This technique must be followed by an author.
[Imperative Level 2]
This technique should be followed by an author.
[Imperative Level 3]
This technique may be followed by an author.
(Adapted by Eric Hansen from the 7 Dec 1998 page authoring guidelines.
Note: I would actually prefer to phrase these differently, but those
changes are immaterial to the argument.)
I submit that these two aspects -- impact and imperative for use -- should
be separated. And the fact that they are not separated in the 7 December
guidelines document is a very serious problem.
Because each priority level inextricably links both impact and imperative
for use, the two aspects cannot vary independently of each other. It makes
no sense to confound the operation of these two variables.
This leads to disastrous results. For example, when a technique is given a
low priority, one does not know whether (a) it is because violation of the
technique was not serious, (b) because the working group believes that is
not imperative (important) that the technique be used, or (c) both (a) and
(b).
This tight coupling of the "impact" and "imperative for use" aspects is a
serious, indeed, a fatal source of bias.
Section 2.B. A Possible Example
Let us consider a situation that might arise. Suppose that an extremely
serious accessibility problem is identified but that the problem is only
poorly addressed by page markup and a full solution is extremely expensive.
Suppose that the high expense of making the change causes the working group
to rate it as having a low imperative for use. The rating system requires
that impact and imperative for use match each other, so the working group
has essentially four choices. First, they can adjust the impact level down
to make it match the imperative for use. Second, they can raise the
imperative for use rating, making it match the impact level. Third, they
can both lower the impact level _and_ raise the imperative for use, thereby
making them match each other. Fourth and finally, they can simply ignore
the accessibility problem.
Section 2.C. An Example From the Document
I believe that this flaw is already causing significant problems.
For example, consider guideline A.8 (Ensure tables have necessary markupț)
is listed as a priority 1 guideline but a note says "[Editor: no P1
techniques here.]." In the A.8 section of the document there is a note
asking "Should this be Pri 1?"
My interpretation of this situation is that the working group judged
guideline A.8 to address a severe (high "impact") problem regarding tables,
such that violation of the guideline would prevent use by one or more
groups. So they gave the guideline a priority 1 rating. Yet there was no
technique that they could heartily endorse and give a priority 1 rating.
Why did they not declare a clear priority 1 to a technique? There are many
possible reasons for this. Perhaps there were concerns about the
feasibility, cost, or effectiveness of the available techniques, so they
chose to give priority 2 or 3 ratings to the techniques. This is just
another example show that the guidelines are using the term "priority" to
represent an impossible amalgam of "impact" and "imperative to use." (Note.
My basic argument is unaffected by the fact that priorities are being
removed from the guidelines.)
PART 3: BIAS AGAINST EXPENSIVE PROBLEMS
Section 3.A. Possible Bias
The foregoing example leads to another major problem, i.e., that the
document may be biased against addressing issues that cannot be solved
inexpensively through page markup. This bias, if correct, is significant
because it appears to have led to other problems that further threaten the
validity of the document.
The guidelines appear biased toward issues that can be fixed _at a low cost
through page markup_ but appears biased against issues that are _more
costly or difficult to solve_, even if the seriousness of the problem might
justify the cost. The document's current stance appears to be that if a
problem that cannot be addressed cheaply through page markup then either it
does not exist or it is relegated to a lower priority. Possible examples of
such issues include language, cultural, sensitivity, legal, economic,
security, and marketing issues. (I do not intend to say that there are no
inexpensive techniques in these domains.)
It may seem strange to refer to an unstated emphasis on inexpensive
solutions as a form of bias. What would be the causes and consequence of
such a bias?
Section 3.B. A Possible Motivator for the Bias
Why would there be a bias against expensive solutions, even when the
solutions might be justified by the seriousness of the accessibility
problem?
Perhaps it boils down the one thing, a concern for "marketing" of the
document. Perhaps the working group is concerned that if Web developers
think that following the accessibility guidelines will be expensive, they
won't pay any attention to the guidelines. Evidence for this marketing
concern is found in a statement in the introduction to section A:
"_Creating pages that transform gracefully is not more costly_, but
requires a different design approach that also makes page compatible with
emerging mobile technologies" (emphasis added). Frankly, this sounds like
marketing hype. The document presents no evidence that this assertion is
true. What may be true, and could perhaps be asserted, is that "Many of the
techniques can be implemented at little or no extra cost, especially at the
design stage of Web development." But to state categorically that the
relevant techniques (evidently A.1 through A.12) will not cost more seems
untenable. If it were mere puffery, it would not be so important. But a
statement like that further paints the guidelines document into a corner.
In order to avoid evidence that might disconfirm the assertion that the
techniques cost no additional money, the document might either downgrade
the imperative for use of expensive techniques or eliminate any reference
to the expensive accessibility problems. This does Web developers a
disservice. By failing to warn them of the existence of the problems, they
are thereby hindered in their ability to address those problems.
Is it wrong to be concerned for the marketing or acceptance of the
document? No. It is absolutely necessary. But I think that marketing of the
document will be most successful if we tell things like they really are and
help Web developers make wise accessibility-related decisions.
Section 3.C. Other Effects
I think that there is evidence that the bias may have affected the document
in other ways.
Consider, for example, the issue of "language" that I have raised
repeatedly. Technique B.3 now addresses the issue partially, but the
technique is relegated to a priority 2 level. Is it possible that the issue
is really a high impact issue (level 1) but that the expense involved in
addressing it the caused it to have very low imperative for use (level 3)
in the eyes of the working group, so a compromise was reached to give it a
level 2 priority?
I have suggested many other issues that could be addressed in the
guidelines (e.g., language, cultural, sensitivity, legal, economic,
marketing, etc.), but, except for partial treatment of the language issue,
to best of my knowledge, the other suggestions have been ignored. Is it
possible that the working group could not see a way to address the issues
cheaply via page markup and that, therefore, they simply chose to act as
though the accessibility problems do not exist?
The bias against expensive, non-markup-related problems may contribute to
another kind of bias. The focus on the inexpensive, non-markup-related
problems places individuals who are deaf or who have language-related
disabilities at a disadvantage relative to individuals who are blind. Most
of the techniques benefit individual who are blind much more than they help
individuals with learning disabilities or deafness. Is this fair? Many of
the accessibility issues that I have raised would benefit individuals who
are deaf or have learning disabilities yet these have been largely ignored.
I wonder if part of the working group's apparent reticence about a
systematic tabulation of impacts is that it will make this fact so
apparent.
Do I have a fundamental problem about the document giving more emphasis to
problems faced by individuals who are blind? No. As long as the scope of
the document and the procedures for producing the ratings are documented
and justified, I have no problem with it.
Do I have a fundamental disagreement with having the document focus on
issues that can be addressed relatively inexpensively through page markup?
No. However, what I do have a deep concern about is a failure to explicitly
state this as well as failure to acknowledge (by name) several other
important and known categories of accessibility-related issues.
PART 4: WHAT IS THE SOLUTION?
I believe that in order to address these problems, the following must be
done.
#1. Uncouple the "impact" ratings from "imperative for use" ratings. They
are separate and must be allowed to vary independently (although, in my
view, the impact ratings inform the generation of the imperative for use
ratings). The impact rating indicates the severity of the _problem_ while
the imperative for use rating is essentially a rating of the importance of
implementing the technique as a _solution_. I believe that to keep the two
ratings together would continually and fatally bias and confuse the
interpretation of the ratings; it would also perpetuate the misconception
that cost considerations are unimportant or irrelevant to the document.
#2. Affirm the existence of other important Web accessibility problems even
if you don't plan to provide specific guidelines or techniques for
addressing them.
#3. Minimize accusations of bias by telling the truth about the document's
scope and limitations. Provide a principled rationale or justification for
the exclusions. I don't think that this is that hard to do. It just has to
be done.
#4. Do not deny the role of cost considerations in the document.
(a) Do not deny that solving some accessibility issues will cost money.
(b) Do not deny that the working group considers matters of cost in
determining the "imperative for use" ratings or even in determining which
problems will be addressed by techniques. Financial considerations
essential for generating "imperative for use" ratings. (The "impact"
ratings would be much less affected by cost considerations.)
#5. Describe how the ratings are produced.
#6. Describe the qualifications of the individuals or group that produced
the ratings. (Note: I believe that failure to do any of #1 through #5
places an excessive burden -- probably an impossible burden -- upon the
expertise and authority of the people making the judgements. I don't think
that it is fair to expect anyone to shoulder that burden.) How were people
with disabilities represented among those developing the ratings?
=============================
Eric G. Hansen, Ph.D.
Development Scientist
Educational Testing Service
ETS 12-R
Rosedale Road
Princeton, NJ 08541
(W) 609-734-5615
(Fax) 609-734-1090
E-mail: ehansen@ets.org
Received on Thursday, 17 December 1998 11:54:09 UTC