Some results from testing forms errors

Hi,

While I was going through the results of an online form errors tests,
the problems that people had using the Palm Beach ballot began surfacing.
The timing was quite surprising and brought out more strongly the
concerns with regard to people making decisions and accurately specifying
what they are.

Web page for forms errors test is at:
       http://members.aol.com/criptrip/test_introl.html


Some background for test:

A couple of months ago someone from Henter-Joyce, a company which
develops screen readers for blind people, contacted me about some of the
experiments I've been doing concerning web technology and blind access.
The discussion ended up being one about the limitations to the help that
access technology can provide.  Basically, while the technology can
usually provide low-level access to information on the screen, the
technology will probably be limited in helping blind people be efficient
in accessing the information in a number of situations.  For example,
how does a person tell a screen reader to move to that object which is
substantially different from the objects around it if the user is
unfamiliar with the web page?

This discussion occurred at the time I was developing an general
information integrator in Java.  While the work is unrelated to
disability, some of the general approaches I've been using could be
applied to the efficiency problem.

One of the challenges that blind people encounter is fixing errors on a
form.  While a screen reader can help the blind user manipulate entries
in a form, the screen reader often cannot identify very efficiently
which form entries have errors.  It can be analogous to a blind person
being put into a room of furniture and being told to fix the broken
table legs.  While the individual most likely can perform the task,
there is probably a difference in efficiency when compared to a sighted
person performing the same task unless there is additional structure.


Methodology:

There are actually two parts to the methodology used on test web page.
One was showing various ways that form errors could be presented.  The
idea was to get some feed-back on preferences in order to develop
approaches to form error presentation which would be more universal.
The qualitative approach used was basically to provide an open-ended
comment field with little direction.

The other part of the methodology was focused on trying to understand in
what ways blind users are less efficient when fixing errors on forms.
The web page used in the test had a mechanism in Javacript which recorded
the time whenever a user changed a form element or selected an error
navigation link.  This mechanism created a log of the actions which
could be examined at a later date for patterns.  When the user selected
the submit button, the activity log was returned along with the other
information.

One concern was that if the form was too long that people wouldn't finish it.
The decision was to make it shorter by including only a few fields without
errors.


Results and Observations

There were 65 forms were returned of which 21 were marked as being from
blind people.  There were 7 pieces of significant email from blind
people.  Three "stream of consciousness" phone calls with blind people
while using the forms.  Two informal visual observations of sighted people
using the forms.  In general, if the various sources of information
were ranked according to usefulness, the order of most useful to less
useful would be something like:

    1.  "stream or consciousness" phone calls and visual observation

    2.  email from people who used the test

    3.  comments in the test

    4.  logs of actions during the test


Two issues about form errors were repeatedly mentioned.  Several people
complained about web pages presenting error messages about forms on a
different page than the forms which frequently happens.  They expressed
positive opinions about the error messages being on the same page as the
forms as used in the test.

One of the discussions that came up was to what degree does having error
messages on a page different from the form imopact users.  This
arrangement might have a greater impact on the efficiency of many blind
form users than sighted form users.  The difference lies in how
information is transmitted to blind users as compared to sighted users.
A sighted persion is given a much larger piece of information at a time
than a blind person, i.e.  an image versus a string of words.  When the
error messages are on a different page than the form, the sighted person
can read an error message, go back to the previous page with the form,
scan the page for the desired field with the error and fix the error.
For many blind users, this arrangement can be much more cumbersome
because they cannot scan over a page.  They may have to start at the
beginning of the page and have the page read to them till they get to
the desired field.  This process is frequently inefficient.

Some blind people can compensate in certain ways.  For example, some
blind people can significantly increase the rate at which a web page is
read which reduces the amount of time needed to get to the field with
the error.  Another way is to use any text search mechanism available,
e.g. search for first name.  Unfortunately, this mechanism is not always
reliable.  For example, suppose that there are two fields on a form for
the first name each of the spouses in a marriage.  Looking for the
appropriate first name field can be a problem.  Another complication
would be if the web page had two forms, each of which had a first name
field.

Some blind people argue that many blind people can be taught to handle these
types of problems.  There are two questions.  One is whether most blind
people can be taught to compensate?  The other is if they are taught to
compensate, how much slower will they be than sighted people if there is
any difference?



The other issue that was brought up a number of times was that the
arrangement of the error messages wasn't consistent.  One purpose of the
test was to try different error message arrangements.  However, the
impression I got was that while it was an annoyance for some sighted
users, it seem to cause more confusion and reduced the efficiency for
various blind users.  Recovering from a confused state seemed to be more of
a problem for various blind users than sighted users.


The web page presented errors with two different approaches.  One
approach was to have links between the error messages so that a user can
jump from error message to error message in a sequential order.  The
other approach was to have the error messages without the links.

The sight users seemed divided about whether the linking was useful or
not.  Some sighted users recognized that the error messages were in
yellow boxes so they could quickly scan for the yellow boxes.  Other
sighted users liked that the links between the error would scroll them
to the next error.  A sighted user commented that she scanned for the
yellow error boxes because the errors were close together, but would
find the links helpful if the errors were farther apart.  Two sighted
users who scanned commented that the linking was annoying.


None of the blind users mentioned that they knew to scan for yellow
boxes so it is not clear that they recognized that the error messages
were in yellow.  There was very favorable response from various blind users to
using links to move from error to error though it was an idea that few
had encountered before.  Trying to find errors on a page can be not very
efficient for many blind users.  Unless the blind person is familiar with
the web page, it can be risky to scan for the word "error" since error
messages may not contain the word error.  One blind user had a comment
similar to one made by a sighted person that the density of the error
messages affected the strategy she used.  Another blind user commented how it
was somewhat confusing because the test form first starts with not using
links and then switches to using links.

Another comment from various blind users was that the error messages
should come before the field with the error rather than after like some
fields on the test form. This arrangement increases the blind user's efficiency
so that they don't have to back up looking for the field and then hear the
error messages again when they move forward.

A comment was made that error messages should be complete enough that
the blind person doesn't have to go hunting around to develop a context
for the error message, e.g. "Please make your selection." is not as
clear as "Please indicate if the nuclear bomb should detonate today or
not."

There were also positive opinions on numbering the error messages.  Many
blind people can have problems knowing how many error messages there are
on a page.  They may have to read the whole form to be sure that they
have found all the error messages.



Recommendations:

(Probably, more testing would be helpful to confirm these recommendatins.)

    1.  redisplay form with error messages instead of error messages on a
        separate page

    2.  put a summary of number of error messages as close to the beginning of
        a web page as possible.  (this will help blind users
	more quickly identify what is happening with the page.)

    3.  put a link from the summary to the first error message area.
        (this can also be useful for sighted people who scan if they
	are unfamiliar with the web page.)

    4.  an error message should come before the field with errors

    5.  error messages should be clearly indicated by a color or an
        image to help people who are scanning.  a combination of color
	and image is probably appropriate in case the user is color-blind

    6.  the error messages should be comprehensive enough so that a
        blind user doesn't have to look around the form for information
	to build a context for the error message

    7.  links should be provided after the error field to navigate
        to the previous or next errors. if the error is the last in the
	form, there should be a link to the submit buttons for the form.

    8.  error messages or links should be numbered e.g. (2/7) , to give a
        context when going through errors


I found that implementing the error linking was actually fairly easily in
both Perl And Java.  Basically, I found that I could handle the problem
by adding four tag types which were:

    <error-summary>
    <error-message-link-target>
    <form-error-check>
    <error-message-link>

For example:

    <error-message-link-target>
    <input name=phone1 size=12>
    <br>
    Message <input name=phone1type type=checkbox value=message>
    Day <input name=phone1type type=checkbox value=day>
    Evening <input name=phone1type type=checkbox value=evening>
    TTD <input name=phone1type type=checkbox value=ttd>
    <form-error-check type=phone field=phone1 phonetyperequired>
    <error-message-link>

In this example, the two tags <form-error-check> and <error-message-link>
can be combined, e.g.:

    <error-message-link type=phone field=phone1 phonetyperequired>

A container structure could have also been used.  Just a matter of personal
preference.




Other observations:

The testing also seemed to indicate that there are some significant
differences between how many sighted and many blind people work with forms
containing error messages (and probably web pages in general).  The sighted
users seemed to have:

    1.  greater ability to scan in a non-linear format

    2.  greater ability to jump over parts of the web page

    3.  greater ability to deduce context

    4.  less need for consistency

    5.  less preference for information density

These observations are pretty informal and need to have more rigorous testing
to see how strong the differences are and to what degree that blind users
will be at disadvantages when compared to sighted users in terms of both
efficiency and accuracy,


The methodology used elicited some interesting qualitative information.
If additional testing were done, it would probably be better to use
two forms so that it would be less confusing.  Also, it would be interesting
to expand the forms to decrease the density of the error messages.
This might create more interesting timing information in the action logs.
Also, instead of one "open-ended" comment area, a series of questions
about various aspects might have been better.


There might be another problem which needs to be addressed in the forms
testing.  An impression I got is that there might be a difference
in how sighted users and blind users perceive lengths of forms, e.g.
what seems like an average length form for a number of sighted users
might be seen as a long form for a number of blind users.  It is
not clear to me how often this might be true.  Also, I'm not sure
what would cause the diffetence in the perceptions.  One conjecture is that
it is harder for many blind users to use forms than sighted users and
the amount of additional effort needed might make the forms appear longer.

The action logs showed another interesting pattern.  A number of the action
logs were very short, i.e. very few actions before submitting.  Only about
10% of the action logs from sighted users were short.  About 50% of the action
logs marked as being from blind users were short.  There can be a technical
reason for this that the default for user type was set to blind user.
If a person submitted the form by key, the user type would default to
blind.  However, sighted people seem less likely to submit by key than
blind users.  If 50% of the blind users did generated short action logs,
this could be indicative of problems that some blind users have with forms.
One problem might be submitting a form by accident or ignorance.  Another
problem might be becoming frustrated more easily and taking a chance to
see what happens.

Given what happened on Palm Beach ballots and also the development of
electronic signatures, there definitely needs to be more research and user
testing about how well a wide range of blind people will be able to be
efficient and accurate in handling forms.  A concern I have is that both
the technology world and the disabled community will put off this type of
analysis until a situation develops which leads to some kind of lawsuit,
e.g. a blind user specifying the wrong choice on a ballot or agreeing
to a contract where some piece of vital information was overlooked.


Scott

Received on Thursday, 16 November 2000 12:57:14 UTC