Re: Pooling test case repositories on the wiki

Hi Carlos,
Thanks for sharing this. Another 700+ test cases is excellent.

W

On Tue, Apr 11, 2017 at 12:07 PM, Shadi Abou-Zahra <shadi@w3.org> wrote:

> Thanks Carlos.
>
> Given how comprehensive the metadata is for these test samples is, I think
> we can extract whatever subset we need in separate JSON files.
>
> Best,
>   Shadi
>
>
>
> On 11-Apr-17 11:24, Carlos A Velasco wrote:
>
>> Hi all,
>>
>> The full set of test cases is available (in 3 versions) in GitHub
>> (Apache license):
>>
>> - https://github.com/webcc/bentoweb-wcag20-test-suite-v1
>> - https://github.com/webcc/bentoweb-wcag20-test-suite-v2
>> - https://github.com/webcc/bentoweb-wcag20-test-suite-v3
>>
>> See also:
>>
>> <http://bentoweb.org/tools.html>
>>
>> On 10/04/17 17:07, Shadi Abou-Zahra wrote:
>>
>>> Hi all,
>>>
>>> In 2006 the (now closed) Evaluation and Repair Tools Working Group (ERT
>>> WG) worked on a "WCAG 2.0 Test Samples Repository":
>>>  - https://www.w3.org/WAI/ER/tests/
>>>
>>> These were contributed by the EC-funded BenToWeb project, which also
>>> developed a comprehensive Test Case Description Language (TCDL). These
>>> Test Samples use a sub-set of the TCDL metadata (documented at the URL
>>> above). We never fully migrated all the Test Samples from the project,
>>> but I'm sure Fraunhofer would be interested in making them available.
>>>
>>> Here is the BenToWeb site for more information:
>>>  - http://www.bentoweb.org/
>>>
>>> Best,
>>>   Shadi
>>>
>>>
>>> On 05-Apr-17 10:59, Wilco Fiers wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> After Monday's discussion I had some ideas on how to proceed with our
>>>> test
>>>> case repository. We've put together a decent size collection of test
>>>> cases
>>>> at this point. But right now they can't be processed automatically
>>>> yet. So
>>>> the question is how do we get to that point.
>>>>
>>>> It seems to me that it would be best if we could let each of these test
>>>> case repos be what they are today. I think we should probably avoid
>>>> forking
>>>> them all and putting them into a new place. That will be way too much
>>>> effort to maintain. So what we could potentially do instead, is that for
>>>> each of these we build a metadata file, that follows a predefined
>>>> structure, so that ATTs can run against those metadata files and run the
>>>> test cases that way. Here's what I'm thinking (written in JSON):
>>>>
>>>> {
>>>>   "name": "axe-core",
>>>>   "version": "2.1.7",
>>>>   "license": "MPL-2.0",
>>>>   "a11y-testcases": [{
>>>>     "url": "
>>>> https://raw.githubusercontent.com/dequelabs/axe-core/master/
>>>> test/integration/rules/image-alt/image-alt.html
>>>>
>>>>
>>>> ",
>>>>     "successCriteria": ["wcag20:text-equiv-all"]
>>>>     "failed": ["#violation1", "#violation2", "#violation3",
>>>> "#violation4"],
>>>>     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
>>>> "#pass6"]
>>>>   }, {
>>>>     ...
>>>>   }]
>>>> }
>>>>
>>>> Thoughts?
>>>>
>>>> W
>>>>
>>>> On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com>
>>>> wrote:
>>>>
>>>> Sorry Alistair for not replying. I missed that part of the email. Upon
>>>>> first reading it I went and added in some links into the wiki page and
>>>>> didn't send an email figuring you were following the page and would get
>>>>> notifications of updates. Obviously, we have access to IBM rules that
>>>>> are
>>>>> not open-source and will share knowledge, if not directly the test
>>>>> cases so
>>>>> did not add them to the list. I am searching the internet for other
>>>>> potential candidates and will add what I find there as well.
>>>>>
>>>>> As for the other email regarding the test case template to follow, I
>>>>> like
>>>>> the 5 parts of the template (copied below so folks don't have to link
>>>>> over
>>>>> to your other email).
>>>>>
>>>>>    1)       Code snippet – element; group of elements; full html
>>>>> element
>>>>>    contents;
>>>>>    2)       Provided by (organisation)
>>>>>    3)       From the test suite of which tool
>>>>>    4)       Expected outcome – is conforming content; is not conforming
>>>>>    content (number of expected issues)
>>>>>    5)       Related WCAG Success Criteria
>>>>>
>>>>>
>>>>> Wondering if we would need the following in the template:
>>>>>
>>>>>    - Title or short name for the test case
>>>>>    - an optional "Notes" section where any known accessibility support
>>>>>    limitations, or any other explanations of the test case would be
>>>>> documented.
>>>>>    - Related WCAG implementation technique (if it is an example of one)
>>>>>
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Mary Jo
>>>>> ------------------------------
>>>>> <http://www.ibm.com/able>
>>>>> *Mary Jo Mueller*
>>>>> Accessibility Standards Program Manager
>>>>> IBM Accessibility, IBM Research, Austin, TX
>>>>> Phone: 512-286-9698 <(512)%20286-9698> | Tie-line: 363-9698
>>>>> *Search for accessibility answers* <http://ibm.biz/a11y-search>
>>>>> <http://www.linkedin.com/e/vgh/2419815/eml-grp-sub/>
>>>>> <http://www.facebook.com/IBMAccessibility>
>>>>> <http://twitter.com/IBMAccess>
>>>>> "If your actions inspire others to dream more, learn more, do more and
>>>>> become more, you are a leader."
>>>>> *~John Quincy Adams*
>>>>>
>>>>> [image: Inactive hide details for Alistair Garrison ---03/22/2017
>>>>> 10:35:45
>>>>> AM---Dear all, I’m in the process of adding a section on “]Alistair
>>>>> Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in the process of
>>>>> adding
>>>>> a section on “Test Case Repositories for WCAG 2.0” to the Tes
>>>>>
>>>>> From: Alistair Garrison <alistair.garrison@ssbbartgroup.com>
>>>>> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
>>>>> Date: 03/22/2017 10:35 AM
>>>>> Subject: Pooling test case repositories on the wiki
>>>>> ------------------------------
>>>>>
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> I’m in the process of adding a section on “Test Case Repositories for
>>>>> WCAG
>>>>> 2.0” to the Testing Resources Wiki (
>>>>> *https://www.w3.org/WAI/GL/task-forces/conformance-testing/
>>>>> wiki/Testing_Resources*
>>>>>
>>>>>
>>>>> <https://www.w3.org/WAI/GL/task-forces/conformance-testing/
>>>>> wiki/Testing_Resources>
>>>>>
>>>>>
>>>>> ).
>>>>>
>>>>> The repositories to be listed in this section will contain examples of
>>>>> what constitutes conformance / non-conformance with WCAG 2.0 SCs, in
>>>>> terms
>>>>> of concrete HTML snippet examples.
>>>>>
>>>>> For example:
>>>>>
>>>>>    1)
>>>>>
>>>>> *https://github.com/dequelabs/axe-core/tree/master/test/inte
>>>>> gration/full*
>>>>>
>>>>>
>>>>>
>>>>> <https://github.com/dequelabs/axe-core/tree/master/test/inte
>>>>> gration/full>
>>>>>
>>>>>
>>>>>       2) *https://github.com/IBMa/Va11yS*
>>>>> <https://github.com/IBMa/Va11yS>
>>>>>
>>>>>
>>>>> Please email me if you have access to, or know of, any further
>>>>> publically
>>>>> available Test Case Repositories? The more repos the merrier!
>>>>>
>>>>> FYI – Here is a link from my previous email relating to Test Case
>>>>> Repositories -
>>>>> *http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb
>>>>> /0007.html*
>>>>> <http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html
>>>>> >
>>>>>
>>>>> Very best regards
>>>>>
>>>>> Alistair
>>>>>
>>>>> ---
>>>>>
>>>>> Alistair Garrison
>>>>> Director of Accessibility Research
>>>>> Level-Access (formerly SSB Bart Group)
>>>>> Visit us online: *Website* <http://www.ssbbartgroup.com/> | *Twitter*
>>>>> <https://twitter.com/SSBBARTGroup> | *Facebook*
>>>>> <https://www.facebook.com/ssbbartgroup> | *LinkedIn*
>>>>> <https://www.linkedin.com/company/355266?trk=tyah> | *Blog*
>>>>> <http://www.ssbbartgroup.com/blog/>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
> --
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> Accessibility Strategy and Technology Specialist
> Web Accessibility Initiative (WAI)
> World Wide Web Consortium (W3C)
>



-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG