Re: Pooling test case repositories on the wiki

Hi everyone,

After Monday's discussion I had some ideas on how to proceed with our test
case repository. We've put together a decent size collection of test cases
at this point. But right now they can't be processed automatically yet. So
the question is how do we get to that point.

It seems to me that it would be best if we could let each of these test
case repos be what they are today. I think we should probably avoid forking
them all and putting them into a new place. That will be way too much
effort to maintain. So what we could potentially do instead, is that for
each of these we build a metadata file, that follows a predefined
structure, so that ATTs can run against those metadata files and run the
test cases that way. Here's what I'm thinking (written in JSON):

{
  "name": "axe-core",
  "version": "2.1.7",
  "license": "MPL-2.0",
  "a11y-testcases": [{
    "url": "
https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html
",
    "successCriteria": ["wcag20:text-equiv-all"]
    "failed": ["#violation1", "#violation2", "#violation3", "#violation4"],
    "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5", "#pass6"]
  }, {
    ...
  }]
}

Thoughts?

W

On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com>
wrote:

> Sorry Alistair for not replying. I missed that part of the email. Upon
> first reading it I went and added in some links into the wiki page and
> didn't send an email figuring you were following the page and would get
> notifications of updates. Obviously, we have access to IBM rules that are
> not open-source and will share knowledge, if not directly the test cases so
> did not add them to the list. I am searching the internet for other
> potential candidates and will add what I find there as well.
>
> As for the other email regarding the test case template to follow, I like
> the 5 parts of the template (copied below so folks don't have to link over
> to your other email).
>
>    1)       Code snippet – element; group of elements; full html element
>    contents;
>    2)       Provided by (organisation)
>    3)       From the test suite of which tool
>    4)       Expected outcome – is conforming content; is not conforming
>    content (number of expected issues)
>    5)       Related WCAG Success Criteria
>
>
> Wondering if we would need the following in the template:
>
>    - Title or short name for the test case
>    - an optional "Notes" section where any known accessibility support
>    limitations, or any other explanations of the test case would be documented.
>    - Related WCAG implementation technique (if it is an example of one)
>
>
> Best regards,
>
> Mary Jo
> ------------------------------
> <http://www.ibm.com/able>
> *Mary Jo Mueller*
> Accessibility Standards Program Manager
> IBM Accessibility, IBM Research, Austin, TX
> Phone: 512-286-9698 <(512)%20286-9698> | Tie-line: 363-9698
> *Search for accessibility answers* <http://ibm.biz/a11y-search>
> <http://www.linkedin.com/e/vgh/2419815/eml-grp-sub/>
> <http://www.facebook.com/IBMAccessibility> <http://twitter.com/IBMAccess>
> "If your actions inspire others to dream more, learn more, do more and
> become more, you are a leader."
> *~John Quincy Adams*
>
> [image: Inactive hide details for Alistair Garrison ---03/22/2017 10:35:45
> AM---Dear all, I’m in the process of adding a section on “]Alistair
> Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in the process of adding
> a section on “Test Case Repositories for WCAG 2.0” to the Tes
>
> From: Alistair Garrison <alistair.garrison@ssbbartgroup.com>
> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
> Date: 03/22/2017 10:35 AM
> Subject: Pooling test case repositories on the wiki
> ------------------------------
>
>
>
> Dear all,
>
> I’m in the process of adding a section on “Test Case Repositories for WCAG
> 2.0” to the Testing Resources Wiki (
> *https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources*
> <https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources>
> ).
>
> The repositories to be listed in this section will contain examples of
> what constitutes conformance / non-conformance with WCAG 2.0 SCs, in terms
> of concrete HTML snippet examples.
>
> For example:
>
>    1)
>       *https://github.com/dequelabs/axe-core/tree/master/test/integration/full*
>       <https://github.com/dequelabs/axe-core/tree/master/test/integration/full>
>       2) *https://github.com/IBMa/Va11yS* <https://github.com/IBMa/Va11yS>
>
>
> Please email me if you have access to, or know of, any further publically
> available Test Case Repositories? The more repos the merrier!
>
> FYI – Here is a link from my previous email relating to Test Case
> Repositories -
> *http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html*
> <http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html>
>
> Very best regards
>
> Alistair
>
> ---
>
> Alistair Garrison
> Director of Accessibility Research
> Level-Access (formerly SSB Bart Group)
> Visit us online: *Website* <http://www.ssbbartgroup.com/> | *Twitter*
> <https://twitter.com/SSBBARTGroup> | *Facebook*
> <https://www.facebook.com/ssbbartgroup> | *LinkedIn*
> <https://www.linkedin.com/company/355266?trk=tyah> | *Blog*
> <http://www.ssbbartgroup.com/blog/>
>
>
>
>
>


-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG