RE: Question about proper use of screen readers in 508 testing

The situation is very simple, but sadly a lot of testers make the same mistake that this team is making. Tools simply point to the possible existence of a WCAG non-conformance. The pass / fail decision should always be made by the tester, based on inspection of the code and the user interface. Tools also fail to identify most issues, so testers need the skills to find those issues themselves. You are right that a lot of testers (perhaps most of them) report based on the output of a tool or assistive technology, perhaps because they don’t understand the code sufficiently well. If that is the case, they need to be trained or replaced.

In the case of the expand/collapse widgets, it sounds like the clickable areas do not have a role of “button” or “link”, which would indeed be a WCAG non-conformance. It can easily be verified by checking the code. The expanded / collapsed state should also be conveyed programmatically. Note that issues like this can easily be found by code inspection - you don’t need a screen reader to find them.
As Patrick said in his reply, there is clearly a training need if the testers think that “button button” is any kind of problem, let alone a critical one. And it’s definitely not a WCAG non-conformance – at most it warrants a comment. Screen reader users have to put up with far worse than that, and they mostly ignore anything they don’t understand.

There are pros and cons to using JAWS rather than NVDA or other screen readers in a WCAG audit. In fact, it’s not necessary to use a screen reader at all, although it can be useful. JAWS used to have the overwhelming market share, but recent surveys suggest that NVDA is now close to parity with it. For WCAG testing purposes I would recommend using NVDA because it gives a relatively unadulterated reflection of the coding, although it has some weirdnesses. By contrast, JAWS uses a lot of heuristics to improve the user experience, so it hides some WCAG non-conformances and can cause some odd behaviours that will not occur with other screen readers.

It is important to understand that a WCAG audit has nothing to do with screen reader behaviour. As Patrick said, the pass / fail decision should be based on WCAG’s normative requirements (with guidance from the supporting non-normative documents). You may choose to also do assistive technology testing, in which case you can report any behaviours you like, but that is different from a WCAG audit, which has a clearly defined scope.

Steve Green
Managing Director
Test Partners Ltd


From: Mike Cleary <mike.cleary@grantsolutions.gov>
Sent: 23 July 2020 16:27
To: w3c-wai-ig@w3.org
Subject: Question about proper use of screen readers in 508 testing

Hello,
I'm new to the discussion list and have a question about how much reliance should be accorded to screen readers like JAWS when reporting accessibility issues.

We have an internal testing team that uses JAWS for 508 testing. They are reporting accessibility issues in cases where JAWS reads all the content on screen, but does not recognize certain expand/collapse widgets as clickable links.

In a different case, they have filed a "critical" bug in cases where a button is read as "button button.." Using the button is no problem; their argument is that the duplicate listing is potentially confusing. I say that's a usability problem, not an accessibility issue and thus not critical.

My concern is that the testers are testing to the tool, not to accessibility guidelines. Am I mistaken? Is there any guidance on how much to rely upon a tool? Is there anything in WCAG 2.0 that speaks to this issue?

Mike

Scrum Master
GrantSolutions.gov