RE: Deque Axe-Core and WCAG mappings

My understanding is that axe-core performs the same tests as the axe browser extension, which does report false positives. We don’t keep records of them, so I can’t produce any evidence right now. If axe-core runs a reduced set of tests compared with the browser extension, then it may well not produce false positives, in which case I apologise.

Depending on the page and page content, axe-core/the axe browser extension(s) can capture as little as 0% of WCAG 2.x issues. Or as much as 100%. It’s different for every website and every page, and the figure decreases progressively as you go through cycles of fixing and retesting.

By contrast, the proportion of WCAG success criteria that a tool can test with 100% reliability does not change – it is the same for all websites and all stages of testing. However, it is very low – typically of the order of 10%. Some success criteria can be partially tested, increasing the coverage to perhaps 20% or so. To get above that, tools start to use heuristics rather than rules, which increases the probability of false positives.

My experience is that the browser extension reports the most issues on websites that are not too badly coded. When the coding is really bad (which is increasingly common, sadly) the tool actually finds fewer issues. For instance, if a tabbed interface has been built using a list of links followed by some content in div elements, the tool will report if one or two of the ARIA attributes are incorrect or missing. However, if all the ARIA attributes are missing, the tool reports no errors because there are no clues that it’s supposed to be a tabbed interface.

BTW, I do know the axe browser extension well because it has been part of our testing methodology for many years. As an external supplier of testing services, we never have access to the development and server environments, so I know less about axe-core other than that it performs the same tests as the browser extension.

Steve


From: John Foliot <john@foliot.ca>
Sent: 29 September 2021 20:56
To: Steve Green <steve.green@testpartners.co.uk>; Wilco Fiers <wilco.fiers@deque.com>
Cc: John Foliot <john@foliot.ca>; Lisette Arocha (US) <lisette.m.arocha@pwc.com>; Wai-Ig <w3c-wai-ig@w3.org>
Subject: Re: Deque Axe-Core and WCAG mappings

Hi Steve,

While I no longer work at Deque, I'd like to seriously push back on your comments regarding false-positives coming from axe-core. Do you have any evidence of axe-core generating false-positives today?

From my 5+ years at Deque, NO FALSE POSITIVES was practically tattooed on the foreheads of the developers who worked on axe-core, and so I reject that comment on the surface, without evidence to the contrary.
Deque's Manifesto states:

  *   Automated accessibility testing rules must have a zero false positive rate
  *   Automated accessibility testing rules must be lightweight and fast
  *   Automated accessibility testing rules must work in all modern browsers
  *   Automated accessibility testing rules must, themselves, be tested automatically
Additionally, axe-core never claimed to "...eliminate the need for any manual testing at all" - in fact, one of Deque's tools (axe Auditor<https://www.deque.com/axe/>) merges both mechanical and manual testing processes and results, and facilitates the additional manual testing process from within the tool (Complete with instructions on how to test, references to WCAG documentation, a standardized reporting language library, and more). Depending on the page and page content however, axe-core/the axe browser extension(s) can capture up to 50% or more of WCAG 2.x issues (and for example, in a content-free CMS template, quite likely even more than 50%).

Finally, I will also note that axe-core CAN BE integrated into production systems (CI/CD integrations<https://github.com/dequelabs/axe-core-gems>), and that was always part of the larger development roadmap for axe-core, so I am not sure why you believe otherwise.

Might I respectfully suggest that if you are going to comment on an auditing tool, that you know the tooling of which you comment on? TIA!

JF

On Wed, Sep 29, 2021 at 1:25 PM Steve Green <steve.green@testpartners.co.uk<mailto:steve.green@testpartners.co.uk>> wrote:
My view is that it’s one huge gap and that axe-core does not eliminate the need for any manual testing at all. It only partially tests most success criteria, and it fully tests so few that it’s not worth trying to work out which they are. And then there are all the false positives that need to be verified manually.

All automated tools are like this. Tools designed for use with continuous integration, like Tenon, give fewer false positives (ideally none) but they find even fewer real faults than other tools (because they rely on rules rather than heuristics).

We only use automated tools as a safety net after we have done a full manual test, in case we missed anything. I would advise others to do the same. We don’t even find them useful for supporting manual testing while we are doing it – in fact we find them a distraction. We use a load of single-purpose bookmarklets and other tools to improve efficiency instead.

Steve Green
Managing Director
Test Partners Ltd


From: John Foliot <john@foliot.ca<mailto:john@foliot.ca>>
Sent: 29 September 2021 18:03
To: Lisette Arocha (US) <lisette.m.arocha@pwc.com<mailto:lisette.m.arocha@pwc.com>>
Cc: Wai-Ig <w3c-wai-ig@w3.org<mailto:w3c-wai-ig@w3.org>>
Subject: Re: Deque Axe-Core and WCAG mappings

Try here: https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md


JF

On Wed, Sep 29, 2021 at 12:45 PM Lisette Arocha (US) <lisette.m.arocha@pwc.com<mailto:lisette.m.arocha@pwc.com>> wrote:
Hi everyone,

I was wondering if anyone has done any work to map out which axe-core rules have good coverage of WCAG success criteria and where the gaps exist for manual testing.

For example, axe-core will pick up where there is no alt attribute but can't determine if the alt text is meaningful, if the image is decorative and should have an empty alt. Additionally, because it only flags the images that have no alt, is there a way to surface what images exist on a site with alt text to help speed up manual testing which would have to check if it's meaningful and/or decorative.

Lisette Maria Arocha
PwC | Manager |  Accessibility & Inclusive Design
Governance, Privacy & Ethics (GP&E) for Products & Technology
Miami | +1 (786) 599 7431
Pronouns: she/her/hers
CPWA, CSM


The information transmitted, including any attachments, is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you received this in error, please contact the sender and delete the material from any computer.



In the event the content of this email includes Tax advice, the content of this email is limited to the matters specifically addressed herein and is not intended to address other potential tax consequences or the potential application of tax penalties to this or any other matter.



PricewaterhouseCoopers LLP is a Delaware limited liability partnership.  This communication may come from PricewaterhouseCoopers LLP or one of its subsidiaries.


--
John Foliot |
Senior Industry Specialist, Digital Accessibility |
W3C Accessibility Standards Contributor |
"I made this so long because I did not have time to make it shorter." - Pascal "links go places, buttons do things"


--
John Foliot |
Senior Industry Specialist, Digital Accessibility |
W3C Accessibility Standards Contributor |
"I made this so long because I did not have time to make it shorter." - Pascal "links go places, buttons do things"