Re: Rules should really cover WCAG Conformance; not WCAG Conformance + Advisory...

Hey Alistair,
The ACT Rules Format doesn't allow mapping to a WCAG success criteria, if
failing that rule wouldn't also fail the SC. A rule that says it tests an
SC, but actually does something else isn't a valid ACT rule. It does
however allow rules that could potentially have false positives. The false
positive scenarios have to be documented in the assumptions, which is the
case for quite a number of the ACT-R rules.

The reality of things is that very few rules are perfectly accurate. We've
known this going in, which is why we've opted not to standardise the rules.
They will need to be updated and refined over time. This is documented in
the ACT Rules Format: https://www.w3.org/TR/act-rules-format/#rule-accuracy

WCAG isn't about getting it right 100% of the time. The rule of thumb going
into WCAG 2 was 80% consistency between experts. I personally think that
put the bar too low. On the other hand, raising it to be 100% seems too
high for me. It definitely isn't what the industry has converged on. It
seems to me that in the automated range, we're somewhere in the 95%+ to
99%+ range.

Making assumptions during testing is the norm. We all do this. What you
think may be advisory, I consider a strong indicator of a conformance
problem. I'm sure there are going to be rules where the reverse is true
too. Finding out which assumptions are agreeable to most of us, and which
ones are unusual is a big part of harmonisation.


As for EU monitoring. I would strongly advise any monitoring body not to
rely a the results of any one rule to determine if an organisation is not
compliant. Accessibility testing isn't a hard science. False positives will
crop up. That's not difficult to deal with, just build some kind of
probability model into the monitoring. If lots of different failures are
reported on different pages, that site is far more likely to be
non-compliant than a site that consistently has 1 error from a single rule
on every page.

W


On Thu, May 2, 2019 at 7:49 PM Mary Jo Mueller <maryjom@us.ibm.com> wrote:

> I know we talked a bit about this in today's ACT TF meeting, but as for
> flagging a rule as advisory - if you don't provide a WCAG success criteria
> in the mapping list, it is not a rule required for conformance. This
> consensus was reached after several months of discussion on how to best
> represent the various rules.
>
> As Wilco said today, I don't think the ACT-R (formerly auto-wcag) intends
> to prioritize rules for advisory techniques or for accessibility best
> practices outside of what is required by WCAG but are simply documenting
> rules that exist in their tools today. This rules development and screening
> process will point out which rules are really advisory and not required.
> The method of publishing them as part of the WCAG materials can help to
> further show whether the rule tests parts of conformance or is instead
> associated with advisory techniques or best practices
>
> Best regards,
>
> Mary Jo
> _____________________________________________
> *Mary Jo Mueller*
> Accessibility Standards Program Manager
> IBM Accessibility Research, Austin, TX
> Phone: 512-286-9698 | Tie-line: 363-9698
>
> "If your actions inspire others to dream more, learn more, do more and
> become more, you are a leader." *~John Quincy Adams*
>
> [image: Inactive hide details for Alistair Garrison ---05/02/2019 08:01:53
> AM---Hi, Internally, we have reviewed the rules developed by]Alistair
> Garrison ---05/02/2019 08:01:53 AM---Hi, Internally, we have reviewed the
> rules developed by ACT-R to date, and have found some to be adv
>
> From: Alistair Garrison <alistair.garrison@levelaccess.com>
> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
> Date: 05/02/2019 08:01 AM
> Subject: Rules should really cover WCAG Conformance; not WCAG Conformance
> + Advisory...
>
> ------------------------------
>
>
>
> Hi,
>
> Internally, we have reviewed the rules developed by ACT-R to date, and
> have found some to be advisory; rather than actually causing a WCAG fail.
>
> I can’t remember if there is a flag for marking a rule as advisory; as
> opposed to required for conformance.
>
> To best support projects such as WAI-Tools, EU monitoring and general
> harmonisation around a WCAG Audit, I think we’ll be aiming to cover only
> those rules required for conformance – leaving out advisory test, at least
> for now.
>
> Also, why are people writing advisory tests? It seems a much lower
> priority than writing rules required for conformance?
>
> Interested to hear others opinions.
>
> All the best
>
> Alistair
>
> ---
>
> Alistair Garrison
> Director of Accessibility Research
> Level Access
>
>
>
>
>

-- 
*Wilco Fiers*
Axe product owner - Co-facilitator WCAG-ACT - Chair ACT-R / Auto-WCAG