Re: Deque Axe-Core and WCAG mappings

Hey folks,

Axe-core product owner stepping into this conversation. Also, all chair
hats I may be wearing from time to time off. Just speaking as Deque product
owner here.

@Lisette, you may find the new "What's left to test" feature of axe
DevTools extension uses. This lists out all the additional testing you may
want to do, after you've completed tests with axe-core and axe Pro:
https://axe.deque.com/coverage-site. I would also ask that you ask further
questions in the axe Community slack channel (
https://accessibility.deque.com/axe-community) The WAI-IG list isn't an
ideal place for questions about specific products or services.

@Steve, I'm happy to hear you find value in using the axe DevTools
extension in your testing process. You are of course right that on some
pages accessibility tools will find no issues, and on others it'll find all
of them. Our own research has shown that on average, 57% of WCAG issues can
be found by axe-core (see
https://accessibility.deque.com/hubfs/Accessibility-Coverage-Report.pdf).
That's based on the total number of issues, rather than the number of
success criteria. Accessibility auditors do tend to think in terms of how
many boxes do I have to check, but I find that talking about totals better
represents the real-world experience of users.

Regarding false positives, I would ask that you open an issue (
https://github.com/dequelabs/axe-core/issues/) if you think you found any.
Seriously. Help us make your workflow better by reporting things. Tools do
have false positives, that's hard to deny. All software has bugs. In the
last year, we had 1 false positive reported for every 5 million downloads.
That's certainly not 0, but it's very rare.

If you commonly find "false positives", the best explanation I can give for
that is that you may have a different understanding of WCAG than we do.
That's always a possibility. You may very well not agree with us that
fallback roles are a problem, or that nested interactive elements are. This
is fairly common, which is why ACT (https://act-rules.github.io/) exists.
It's an open group, so if anyone is interested to help shape how WCAG
testing is shaped, including how axe-core and quite a few other tools
handle WCAG edge cases, you are welcome to join.

Kind regards,

On Thu, Sep 30, 2021 at 6:41 AM Steve Green <steve.green@testpartners.co.uk>
wrote:

> My understanding is that axe-core performs the same tests as the axe
> browser extension, which does report false positives. We don’t keep records
> of them, so I can’t produce any evidence right now. If axe-core runs a
> reduced set of tests compared with the browser extension, then it may well
> not produce false positives, in which case I apologise.
>
>
>
> Depending on the page and page content, axe-core/the axe browser
> extension(s) can capture as little as 0% of WCAG 2.x issues. Or as much as
> 100%. It’s different for every website and every page, and the figure
> decreases progressively as you go through cycles of fixing and retesting.
>
>
>
> By contrast, the proportion of WCAG success criteria that a tool can test
> with 100% reliability does not change – it is the same for all websites and
> all stages of testing. However, it is very low – typically of the order of
> 10%. Some success criteria can be partially tested, increasing the coverage
> to perhaps 20% or so. To get above that, tools start to use heuristics
> rather than rules, which increases the probability of false positives.
>
>
>
> My experience is that the browser extension reports the most issues on
> websites that are not too badly coded. When the coding is really bad (which
> is increasingly common, sadly) the tool actually finds fewer issues. For
> instance, if a tabbed interface has been built using a list of links
> followed by some content in div elements, the tool will report if one or
> two of the ARIA attributes are incorrect or missing. However, if all the
> ARIA attributes are missing, the tool reports no errors because there are
> no clues that it’s supposed to be a tabbed interface.
>
>
>
> BTW, I do know the axe browser extension well because it has been part of
> our testing methodology for many years. As an external supplier of testing
> services, we never have access to the development and server environments,
> so I know less about axe-core other than that it performs the same tests as
> the browser extension.
>
>
>
> Steve
>
>
>
>
>
> *From:* John Foliot <john@foliot.ca>
> *Sent:* 29 September 2021 20:56
> *To:* Steve Green <steve.green@testpartners.co.uk>; Wilco Fiers <
> wilco.fiers@deque.com>
> *Cc:* John Foliot <john@foliot.ca>; Lisette Arocha (US) <
> lisette.m.arocha@pwc.com>; Wai-Ig <w3c-wai-ig@w3.org>
> *Subject:* Re: Deque Axe-Core and WCAG mappings
>
>
>
> Hi Steve,
>
>
>
> While I no longer work at Deque, I'd like to seriously push back on your
> comments regarding false-positives coming from axe-core. *Do you have any
> evidence of axe-core generating false-positives today*?
>
>
>
> From my 5+ years at Deque, NO FALSE POSITIVES was practically tattooed on
> the foreheads of the developers who worked on axe-core, and so I reject
> that comment on the surface, without evidence to the contrary.
>
> Deque's Manifesto states:
>
>
>    - Automated accessibility testing rules must have a zero false
>    positive rate
>    - Automated accessibility testing rules must be lightweight and fast
>    - Automated accessibility testing rules must work in all modern
>    browsers
>    - Automated accessibility testing rules must, themselves, be tested
>    automatically
>
> Additionally, axe-core never claimed to "...*eliminate the need for any
> manual testing at all*" - in fact, one of Deque's tools (axe Auditor
> <https://www.deque.com/axe/>) merges both mechanical and manual testing
> processes and results, *and facilitates the additional manual testing
> process from within the tool *(Complete with instructions on how to test,
> references to WCAG documentation, a standardized reporting language
> library, and more). Depending on the page and page content however,
> axe-core/the axe browser extension(s) can capture up to 50% or more of WCAG
> 2.x issues (and for example, in a content-free CMS template, quite likely
> even more than 50%).
>
> Finally, I will also note that axe-core CAN BE integrated into production
> systems (CI/CD integrations <https://github.com/dequelabs/axe-core-gems>),
> and that was always part of the larger development roadmap for axe-core, so
> I am not sure why you believe otherwise.
>
>
>
> Might I respectfully suggest that if you are going to comment on an
> auditing tool, that you know the tooling of which you comment on? TIA!
>
>
>
> JF
>
>
>
> On Wed, Sep 29, 2021 at 1:25 PM Steve Green <
> steve.green@testpartners.co.uk> wrote:
>
> My view is that it’s one huge gap and that axe-core does not eliminate the
> need for any manual testing at all. It only partially tests most success
> criteria, and it fully tests so few that it’s not worth trying to work out
> which they are. And then there are all the false positives that need to be
> verified manually.
>
>
>
> All automated tools are like this. Tools designed for use with continuous
> integration, like Tenon, give fewer false positives (ideally none) but they
> find even fewer real faults than other tools (because they rely on rules
> rather than heuristics).
>
>
>
> We only use automated tools as a safety net after we have done a full
> manual test, in case we missed anything. I would advise others to do the
> same. We don’t even find them useful for supporting manual testing while we
> are doing it – in fact we find them a distraction. We use a load of
> single-purpose bookmarklets and other tools to improve efficiency instead.
>
>
>
> Steve Green
>
> Managing Director
>
> Test Partners Ltd
>
>
>
>
>
> *From:* John Foliot <john@foliot.ca>
> *Sent:* 29 September 2021 18:03
> *To:* Lisette Arocha (US) <lisette.m.arocha@pwc.com>
> *Cc:* Wai-Ig <w3c-wai-ig@w3.org>
> *Subject:* Re: Deque Axe-Core and WCAG mappings
>
>
>
> Try here:
> https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md
>
> JF
>
>
>
> On Wed, Sep 29, 2021 at 12:45 PM Lisette Arocha (US) <
> lisette.m.arocha@pwc.com> wrote:
>
> Hi everyone,
>
>
>
> I was wondering if anyone has done any work to map out which axe-core
> rules have good coverage of WCAG success criteria and where the gaps exist
> for manual testing.
>
>
>
> For example, axe-core will pick up where there is no alt attribute but
> can't determine if the alt text is meaningful, if the image is decorative
> and should have an empty alt. Additionally, because it only flags the
> images that have no alt, is there a way to surface what images exist on a
> site with alt text to help speed up manual testing which would have to
> check if it's meaningful and/or decorative.
>
>
> *Lisette* *Maria* *Arocha*
>
> PwC | Manager |  Accessibility & Inclusive Design
>
> Governance, Privacy & Ethics (GP&E) for Products & Technology
>
> Miami | +1 (786) 599 7431
>
> Pronouns: she/her/hers
>
> CPWA, CSM
>
>
>
> The information transmitted, including any attachments, is intended only
> for the person or entity to which it is addressed and may contain
> confidential and/or privileged material. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient
> is prohibited, and all liability arising therefrom is disclaimed. If you
> received this in error, please contact the sender and delete the material
> from any computer.
>
>
>
> In the event the content of this email includes Tax advice, the content of
> this email is limited to the matters specifically addressed herein and is
> not intended to address other potential tax consequences or the potential
> application of tax penalties to this or any other matter.
>
>
>
> PricewaterhouseCoopers LLP is a Delaware limited liability partnership.
> This communication may come from PricewaterhouseCoopers LLP or one of its
> subsidiaries.
>
>
>
>
> --
>
> *John Foliot* |
> Senior Industry Specialist, Digital Accessibility |
> W3C Accessibility Standards Contributor |
>
> "I made this so long because I did not have time to make it shorter." -
> Pascal "links go places, buttons do things"
>
>
>
>
> --
>
> *John Foliot* |
> Senior Industry Specialist, Digital Accessibility |
> W3C Accessibility Standards Contributor |
>
> "I made this so long because I did not have time to make it shorter." -
> Pascal "links go places, buttons do things"
>


-- 
*Wilco Fiers*
Axe-core product owner - Facilitator ACT Task Force - Co-chair ACT-Rules