Re: Test Suites

Hi Alistair,

It's a fairly lightweight approach, but I think it might be problematic if
we only include how many issues to expect. I'd much rather that for each
element that is passed or failed, we include a pointer so we can be sure
that tools didn't misidentify elements that were in error. Just because the
error count was the same, doesn't mean it found the errors that it was
expected to find.

As for indicating the success criteria that go with it. What we might do is
include a list of criteria that it might apply under, and only consider it
an error if the tool comes up with a criterion that isn't on that list.
That way if one tool reports something in 1.1.1, and another in 4.1.2, and
both are on the list of criteria that an error might go in, those would be
valid, but reporting it for 2.2.2 wouldn't be.

I think this is a nice light weight way to start with. I we'll also need to
consider what this would look like with rules. I imagine that in addition
to listing criteria, we'd include which rule(s?) should generate that
particular result.

Wilco

On Wed, Feb 8, 2017 at 4:00 PM, Alistair Garrison <
alistair.garrison@ssbbartgroup.com> wrote:

> Dear All,
>
>
>
> Looking at the simplest template for codifying conforming / non-conforming
> examples of code; which can easily be consumed by any other test suite /
> testing framework – I’d like to propose the following:
>
>
>
> 1)       Code snippet – element; group of elements; full html element
> contents;
>
> 2)       Provided by (organisation)
>
> 3)       From the test suite of which tool
>
> 4)       Expected outcome – is conforming content; is not conforming
> content (number of expected issues)
>
> 5)       Related WCAG Success Criteria
>
>
>
> What are your thoughts / comments?
>
>
>
> Very best regards
>
>
>
> Alistair
>
>
>
> ---
>
>
>
> Alistair Garrison
>
> Director of Accessibility Research
>
> SSB Bart Group
>
> Visit us online: Website <http://www.ssbbartgroup.com/> | Twitter
> <https://twitter.com/SSBBARTGroup> | Facebook
> <https://www.facebook.com/ssbbartgroup> | LinkedIn
> <https://www.linkedin.com/company/355266?trk=tyah> | Blog
> <http://www.ssbbartgroup.com/blog/>
>
>
>
>
>



-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG