Re: Framework Doc - Accuracy Benchmarking

Hi Alistair,

I don't disagree that the benchmark will be complicated. I don't think it
can be done without comparing to manually tested pages. Which will require
us to get access to manually test data. Gathering all that data ourselves
would be pretty impractical, But there are a lot of organisations that have
this data. So I'd say that our best bet would be to see who is able to
share their accessibility data, so that we can build off of that.

As for your suggestion to have test suites. I expect this will be part of
the requirement for a rule. I want to clearly distinguish between *testing
the rule* itself, which is what the benchmark is for, and *testing the
implementation of the rule*, which is what a test suite would be for.

Thoughts?

Wilco

On Wed, Nov 16, 2016 at 10:43 AM, Alistair Garrison <
alistair.garrison@ssbbartgroup.com> wrote:

> Dear All,
>
>
>
> After reading through the Framework Document - https://w3c.github.io/wcag-
> act/act-framework.html#quality-updates, I would say that the Accuracy
> Benchmarking concept might be tricky as it is described.
>
>
>
> The section reads – “Measuring this accuracy, not just on test data, but
> on pages and components that the rules would actually be applied to, is
> important to give users of the test results confidence in the accuracy of
> this data.”
>
>
>
> The question is – how do you tell if the test is working properly in a
> live page, without first looking at the relevant code in the page (and
> assessing it – possibly manually, or with another tool)?
>
>
>
> I would propose we concentrate on creating test suites for each of the
> tests – with an ever growing number of clearly specified real-world edge
> cases that we are informed about by the testing community.
>
>
>
> Very best regards
>
>
>
> Alistair
>
>
>
> ---
>
>
>
> Alistair Garrison
>
> Senior Accessibility Engineer
>
> SSB Bart Group
>
>
>
>
>
>
>
>
>
>
>



-- 
*Wilco Fiers* - Senior Accessibility Engineer