Re: AW: Using AT for evaluation

I agree with Kerstin. We should "highly recommend" the use of an AT for 
validation. 

We require our testers to test with the JAWS screen reader for 3 reasons:
To ensure that our applications can be used by persons who are blind. We 
use JAWS as an end user application.
To help validate our ARIA implementation. Freedom Scientific has made 
great strides in supporting ARIA in the last 2 releases. 
We use JAWS as a tool to access structure and elements on the page to help 
us quickly and accurately verify that they are coded properly. We use JAWS 
to test ARIA landmark navigation, ensure form fields and links have unique 
and appropriate labels, ensure headings are properly defined, ensure 
controls have appropriate name, state, role and value, etc. 

We do not currently require the use of an SR during test. 

Moe Kraft

Maureen E Kraft, Accessibility Test Consultant Lead
Human Ability & Accessibility Center, IBM Research
www.ibm.com/able & w3.ibm.com/able
maureen_kraft@us.ibm.com,(978) 899-3114, T/L 276-3114



From:   "Kerstin Probiesch" <k.probiesch@googlemail.com>
To:     "'Shadi Abou-Zahra'" <shadi@w3.org>, "'RichardWarren'" 
<richard.warren@userite.com>, 
Cc:     "'Elle'" <nethermind@gmail.com>, "'Detlev Fischer'" 
<detlev.fischer@testkreis.de>, <public-wai-evaltf@w3.org>
Date:   04/04/2012 05:19 PM
Subject:        AW: Using AT for evaluation



Hi all,

as already written: I also think that "optional" is not sufficient and
concur the proposal "highly recommended", probably like this: "testing 
with
AT is highly recommended especially in the following cases

# forms and (complex) data tables: testing at least with two common 
browsers
and two commons SRs
# JavaScript...
# ...

I propose to distinct between web and closed environments. For closed
environments I think that testing with AT must be done and that
http://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance.html#uc-accessibility-


support-head will give us a lot of input. If there are closed environments
where just one browser is allowed and probably just one or two SR than I
think it must be tested with this browser and the those SR. If it would 
just
optional (even when we would say "highly recommended" it would be 
optional)
I fear that it wouldn't be done. In closed environments it is much more
important than in the web where a user probably has alternatives. In
Intranets there are no alternatives.

The question is how to handle the repair behavior. Ok. One shouldn't rely
upon the repair behavior in the web. But what is with the repair behavior 
in
closed environments? For example: one browser is allowed and one SR and an
app works because of the repair behavior. Is it then accessibility 
supported
or not? If it works, it works? One could say that it works just because of 
a
repair behavior, but would it be realistic that a company spend time and
money in changing something which already works even when it is just a
repair behavior?

Best

Kerstin

 

> -----Ursprüngliche Nachricht-----
> Von: Shadi Abou-Zahra [mailto:shadi@w3.org]
> Gesendet: Mittwoch, 4. April 2012 20:05
> An: RichardWarren
> Cc: Elle; Detlev Fischer; public-wai-evaltf@w3.org
> Betreff: Re: Using AT for evaluation
> 
> Hi Richard,
> 
> While WCAG 2.0 may not specifically say that content must be tested
> with
> assistive technology, it clearly requires content to work with
> assistive
> technology:
> 
>  From <http://www.w3.org/TR/WCAG20/#new-terms> [[
> Using a technology in a way that is accessibility supported means that
> it works with assistive technologies (AT) and the accessibility
> features
> of operating systems, browsers, and other user agents. Technology
> features can only be relied upon to conform to WCAG 2.0 success
> criteria
> if they are used in a way that is "accessibility supported".]]
> 
> At the same time, "assistive technologies (AT) and the accessibility
> features of operating systems, browsers, and other user agents" have
> bugs and other peculiarities that a developer cannot always be held
> accountable to have to work around.
> 
> I think Elle's previous suggestion of using scenario-like approaches
> seems promising but the question remains: how to demonstrate that a
> website works with assistive technology?
> 
> Best,
>    Shadi
> 
> 
> On 4.4.2012 18:50, Elle wrote:
> > All:
> >
> > I can definitely see Detlev's point, and I concur about "highly
> > recommended" as an addition.  However, I still request that we not
> limit
> > ourselves or our audience to simply a screen reader.
> >
> >
> >
> > Thanks,
> > Elle
> >
> >
> >
> > On Wed, Apr 4, 2012 at 12:46 PM, RichardWarren
> > <richard.warren@userite.com>wrote:
> >
> >> Aah,
> >>
> >> Detlev has a point. Our methodology is to check for compliance with
> the
> >> W3C guidelines, it is not, in and of itself, to test for universal
> >> accessibility. The guidelines do not require, as an essential
> component,
> >> that the site be tested with a screen reader. On that basis we do
> not have
> >> the right to insist on using a screen reader (of whatever flavour).
> As
> >> Detlev says it is possible to check for conformance without using a
> screen
> >> reader, and not all Screen readers are compliant themselves.
> >>
> >> HOWEVER I feel that making the use of a screen reader "optional" is
> not
> >> sufficient. I sense the feeling of the group is that screen reader
> testing
> >> is important - so I wonder if we can call it's use  "recommended",
> or even
> >> "highly recommended" rather than "optional".
> >>
> >> Richard
> >>
> >> -----Original Message----- From: Detlev Fischer
> >> Sent: Wednesday, April 04, 2012 4:40 PM
> >> To: public-wai-evaltf@w3.org
> >>
> >> Cc: Eval TF
> >> Subject: Re: AW: Using AT for evaluation
> >>
> >> Hi list,
> >>
> >> I disagree with the proposed mandatory requirement to use a screen
> >> reader as part of WCAG-EM, for several reasons:
> >>
> >> 1. Inconclusive result from any one SR use
> >> 2. Difficulty of separating conformant behaviour from SR repair
> >> behaviour
> >> 3. Much higher professional prerequisite for conducting evaluations
> >>
> >> Let me quickly elaborate these three points.
> >>
> >> 1) Inconclusive result from any one SR use. Even for well known
> >> semantic constructs such as forms with legends or tables, SR
> >> implementation varies so widely that a well constructed form or
> table
> >> might be OK in one SR and can pose problems in another. What must
> >> surely count is whether content has been implemented correctly, not
> if
> >> some AT is able to use it as intended. For most cases, checks exist
> if
> >> semantics have been implemented according to spec or technique that
> do
> >> not require the use of a SR.
> >> To take the example of correct pronunciation of words mentioned by
> >> several: If a word is pronounced correctly will depend on the
> >> inclusion of the word in the SR dictionary and also, the options set
> >> by individual users. It will vary a lot across SR makes and
> versions.
> >> (I have met more than one SR user who willingly turned off other
> >> languages because he/he opted for speedy delivery over correct
> choice
> >> of language).
> >> Finally, leaving it open which SR to use makes results rather
> >> inconclusive. I think we cannot and should not mandate, say, the use
> >> of NVDA simply because it is free of charge. If anything, it should
> be
> >> the most frequently used SR (often JAWS) which costs quite a bit
> (see
> >> point 3).
> >>
> >> 2) Difficulty of separating conformant behaviour from SR repair
> >> behaviour. SR show a wide range of behaviour towards conformant and
> >> badly formatted content, which makes it difficult to rely on any one
> >> outcome in a conformance evaluation. Since the web is such a mess on
> >> the whole, SR are good in coping with that mess, but to varying
> >> degrees. Take a text rendered using Cufon. Jaws will read through it
> >> OK, other SRs will stutter, voiceover just reads image - image -
> >> image. So how does any one of these results help if we do not intend
> >> to cover a range (which I believe is impossible due to time and
> budget
> >> constraints)?
> >>
> >> 3) Much higher professional prerequisites for conducting
> evaluations.
> >> Requiring the use of a SR without making sure that it is used
> >> proficiently will lead to dubious results. SR have shortcuts and
> modes
> >> of operation that help expert users make sense of content. So if SR
> >> use becomes a mandatory part of the evaluation there should be a
> >> requirement that evaluators know their SR well enough to have
> >> meaningful results. This puts the bar a lot higher for folks who
> look
> >> for evaluation guidance but are not prepared for the time and effort
> >> to learn how to use a SR adequately.
> >> And adequate use will be difficult to define, especially if we do
> not
> >> mandate a SR.
> >>
> >> Having said all that, there are some points which are tricky to
> >> evaluate without a SR - for example, checking that when inserting
> >> content via DOM and setting the focus via a script, the SR focus
> will
> >> indeed move to the content inserted. But this touches on the
> question
> >> whether we should rely strongly on any observed actual AT
> >> implementation behaviour at any given point in time. If AT does not
> >> yet manage to deal adequately with correct code, the alternative
> would
> >> be to check whether implementation is done according to spec and put
> >> the burden on AT to catch up. Of course, checking for 'correct code'
> >> is difficult without being s scripting expert, it also raises the
> bar.
> >> This is a tricky one, and I am not really sure what's best here.
> >>
> >> Regards,
> >> Detlev
> >>
> >>
> >> On 4 Apr 2012, at 15:57, Michael S Elledge wrote:
> >>
> >>   Hi Everyone--
> >>>
> >>> My preference would be to require the use of a screen reader in
> >>> conducting an evaluation, either with or without a person with
> >>> disabilities. The option, in my mind, would be having it done by a
> person
> >>> with disabilities.
> >>>
> >>> With the availability of free screen readers like NVDA, testers
> ought to
> >>> be able to incorporate it in testing without incurring
> unreasonable costs.
> >>> I realize this falls short of the ideal, which  is evaluation by a
> variety
> >>> of people with different disabilities,  but we've found it to be
> critical
> >>> in our testing for discovering  issues (such as pronunciation or
> >>> functionality) that would otherwise  be missed.
> >>>
> >>> I think another AT frequently used by persons with disabilities,
> screen
> >>> enlargers like ZoomText, may be unnecessary in the testing
> methodology, so
> >>> long as other methods are used to review an enlarged  screen. I
> bring that
> >>> up for discussion, because others may not agree  with me.
> >>>
> >>> Best regards,
> >>>
> >>> Mike
> >>>
> >>> On 4/2/2012 8:17 PM, Vivienne CONWAY wrote:
> >>>
> >>>>
> >>>> HI all
> >>>>
> >>>> I always use screen readers and am wary when clients say that they
> >>>>   don't need testing with AT because they don't have any disabled
> users.
> >>>>   We never know the extent of employees' limitations - they  don't
> have to
> >>>> disclose everything.  Also, it often happens that  someone suffers
> an
> >>>> injury or illness while in employment only to  find that they
> can't use the
> >>>> system now that worked for them  previously.  The old 1 in 5
> thought
> >>>> regarding disabilities applies  to those in employment as to the
> general
> >>>> public in their use of a  website.
> >>>>
> >>>> So... I would suggest that AT (at least the screen reader) be
> strongly
> >>>> encouraged for all website/application testing.
> >>>>
> >>>>
> >>>> Regards
> >>>>
> >>>> Vivienne L. Conway, B.IT(Hons), MACS CT
> >>>> PhD Candidate&  Sessional Lecturer, Edith Cowan University, Perth,
> W.A.
> >>>> Director, Web Key IT Pty Ltd.
> >>>> v.conway@ecu.edu.au
> >>>> v.conway@webkeyit.com
> >>>> Mob: 0415 383 673
> >>>>
> >>>> This email is confidential and intended only for the use of the
> >>>> individual or entity named above. If you are not the intended
> recipient,
> >>>> you are notified that any dissemination, distribution or  copying
> of this
> >>>> email is strictly prohibited. If you have received  this email in
> error,
> >>>> please notify me immediately by return email  or telephone and
> destroy the
> >>>> original message.
> >>>> ______________________________**__________
> >>>> From: RichardWarren [richard.warren@userite.com]
> >>>> Sent: Tuesday, 3 April 2012 2:28 AM
> >>>> To: Kerstin Probiesch
> >>>> Cc: 'Eval TF'
> >>>> Subject: Re: AW: Using AT for evaluation
> >>>>
> >>>> Dear Kerstin,
> >>>>
> >>>> I don't object too much if a "real user" (ie blind person) doesn't
> use a
> >>>> screen reader to test the site - the most important thing is that
> a
> >>>> screen
> >>>> reader is used. Only a screen reader's audio output would
> demonstrate
> >>>> misspelled words and phone numbers (thanks Denis). You would not
> test
> >>>> without using a visual browser so you should also use an audio
> browser.
> >>>>
> >>>> For an enclosed environment, such as an Intranet, it could be
> possible
> >>>> to
> >>>> exclude testing with certain AT - only IF you know that the
> technology
> >>>> will
> >>>> not be required. However this would then mean that the Intranet
> could
> >>>> never
> >>>> be used by such a disabled person and could breach employment
> >>>> legislation.
> >>>>
> >>>> Richard
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: Kerstin Probiesch
> >>>> Sent: Monday, April 02, 2012 3:51 PM
> >>>> To: 'Denis Boudreau' ; 'RichardWarren'
> >>>> Cc: 'Eval TF'
> >>>> Subject: AW: Using AT for evaluation
> >>>>
> >>>> Hi Richard, Denis, all,
> >>>>
> >>>> I also think that test with "real" (at least screen reader) users
> are
> >>>> important and that we should strongly recommend it but leave it
> >>>> optional. As
> >>>> I remember the discussion on our last telco there are two aspects:
> >>>>
> >>>> - testing with AT and
> >>>> - accessibility supported
> >>>>
> >>>> I think we have an intersection but also other aspects like: are
> >>>> technologies like PDF and Flash accessibility supported? Depending
> upon
> >>>> the
> >>>> answer it will have probably different consequences for our
> methodology.
> >>>> Also different use cases like internet and intranet (especially
> when it
> >>>> comes to scripting for JAWS or other screen readers in closed
> >>>> environments)
> >>>> might have an impact. I'm thinking about if we could find for the
> tests
> >>>> of
> >>>> intranets something better than just "optional" without reducing
> the
> >>>> audience of our methodology in whole.
> >>>>
> >>>> Best
> >>>>
> >>>> Kerstin
> >>>>
> >>>>
> >>>> Von: Denis Boudreau
> [mailto:dboudreau@**accessibiliteweb.com<dboudreau@accessibiliteweb.com
> >
> >>>> ]
> >>>> Gesendet: Montag, 2. April 2012 15:40
> >>>> An: RichardWarren
> >>>> Cc: Eval TF
> >>>> Betreff: Re: Using AT for evaluation
> >>>>
> >>>> Hi Richard,
> >>>>
> >>>> I would also like to weigh in with Richard here. All too often,
> screen
> >>>> reader testing is considered a luxury that can be done without. I
> am
> >>>> one of
> >>>> those who think that an evaluation cannot be considered complete
> unless
> >>>> some
> >>>> screen reader testing has been conducted - and ideally, not only
> by a
> >>>> developer, but really by a "real" end user with a visual
> impairment,
> >>>> using
> >>>> the assistive technology regularly. The same could be said of
> other  end
> >>>> user
> >>>> using other tools for other disabilities or limitations, but at
> the  very
> >>>> least, screen readers.
> >>>>
> >>>> There are always things that are brought up with AT testing that
> cannot
> >>>> be
> >>>> flagged using only a checklist. Some of the things that come to
> mind are
> >>>> links used for buttons that really should be coded as<button>, an
> >>>> overwhelming number of heading elements in a page (big menus and
> fat
> >>>> footers
> >>>> anyone?) or quite obviously, any script that opens up or reveals
> >>>>   content in
> >>>> a page. I recently had big surprises simply with phone number
> formats
> >>>> and
> >>>> how screen readers read them. That was another real eye opener (no
> pun
> >>>> intended).
> >>>>
> >>>> This is why I tend to follow this pattern personally:
> >>>>
> >>>> * testing the web page with a screen reader
> >>>> * using an automatic checker for basic problems
> >>>> * running manual testing to complete the audit
> >>>>
> >>>> And whenever I am being offered the budget to do so, calling in a
> >>>> visually
> >>>> impaired friend or two who can push those tests much further that
> my
> >>>> sighted
> >>>> self can push them.
> >>>>
> >>>> /Denis
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 2012-03-29, at 6:48 PM, RichardWarren wrote:
> >>>>
> >>>>
> >>>> First – sorry I missed the last half of the teleconference –
> system
> >>>> crash.
> >>>>
> >>>> I wish to add to the discussion on using AT in evaluation. I
> believe it
> >>>> is
> >>>> important to use a screen reader at the very least before
> completing an
> >>>> evaluation. We do the normal stuff first (it is not fair to ask a
> blind
> >>>> user
> >>>> to struggle if we already know that the site is impossible for
> them).
> >>>> But as
> >>>> soon as we are happy that a site is reasonably good we always ask
> >>>> someone to
> >>>> check with their screen reader. Most times their comments re-
> inforce
> >>>> what we
> >>>>
> >>>> have found (often with better phrasing<G>). But just occasionally
> they
> >>>> find
> >>>> something that our other systems do not pick up. For example the
> word
> >>>> “accesskeys” sound completely Russian unless it is written “access
> >>>>   keys”, or
> >>>> “access-keys”.
> >>>>
> >>>> I strongly believe that the audio output needs to be checked
> properly.
> >>>> If
> >>>> you look at our outline procedure sent to this list on 26 Feb you
> will
> >>>> see
> >>>> that we find someone who is new to the site to use a screen
> reader.  This
> >>>> approach gives us a high level of confidence in our final
> evaluation.
> >>>>
> >>>> Richard
> >>>>
> >>>> Technical Manager
> >>>> Website Auditing Limited
> >>>> http://www.userite.com
> >>>>
> >>>> This e-mail is confidential. If you are not the intended recipient
> you
> >>>> must not disclose or use the information contained within. If  you
> have
> >>>> received it in error please return it to the sender via  reply e-
> mail and
> >>>> delete any record of it from your system. The  information
> contained within
> >>>> is not the opinion of Edith Cowan  University in general and the
> University
> >>>> accepts no liability for  the accuracy of the information
> provided.
> >>>>
> >>>> CRICOS IPC 00279B
> >>>>
> >>>>
> >>>>
> >>>>
> >>> --
> >>> Michael S. Elledge
> >>> Associate Director
> >>> Usability/Accessibility Research and Consulting
> >>> Michigan State University
> >>> Kellogg Center
> >>> 219 S. Harrison Rd Room 93
> >>> East Lansing, MI  48824
> >>> 517-353-8977
> >>>
> >>
> >> --
> >> Detlev Fischer
> >> testkreis - das Accessibility-Team von feld.wald.wiese
> >> c/o feld.wald.wiese
> >> Borselstraße 3-7 (im Hof)
> >> 22765 Hamburg
> >>
> >> Tel   +49 (0)40 439 10 68-3
> >> Mobil +49 (0)1577 170 73 84
> >> Fax   +49 (0)40 439 10 68-5
> >>
> >> http://www.testkreis.de

> >> Beratung, Tests und Schulungen für barrierefreie Websites
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> 
> --
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/

> Activity Lead, W3C/WAI International Program Office
> Evaluation and Repair Tools Working Group (ERT WG)
> Research and Development Working Group (RDWG)