Re: Pooling test case repositories on the wiki

Thanks Carlos.

Given how comprehensive the metadata is for these test samples is, I 
think we can extract whatever subset we need in separate JSON files.

Best,
   Shadi


On 11-Apr-17 11:24, Carlos A Velasco wrote:
> Hi all,
>
> The full set of test cases is available (in 3 versions) in GitHub
> (Apache license):
>
> - https://github.com/webcc/bentoweb-wcag20-test-suite-v1
> - https://github.com/webcc/bentoweb-wcag20-test-suite-v2
> - https://github.com/webcc/bentoweb-wcag20-test-suite-v3
>
> See also:
>
> <http://bentoweb.org/tools.html>
>
> On 10/04/17 17:07, Shadi Abou-Zahra wrote:
>> Hi all,
>>
>> In 2006 the (now closed) Evaluation and Repair Tools Working Group (ERT
>> WG) worked on a "WCAG 2.0 Test Samples Repository":
>>  - https://www.w3.org/WAI/ER/tests/
>>
>> These were contributed by the EC-funded BenToWeb project, which also
>> developed a comprehensive Test Case Description Language (TCDL). These
>> Test Samples use a sub-set of the TCDL metadata (documented at the URL
>> above). We never fully migrated all the Test Samples from the project,
>> but I'm sure Fraunhofer would be interested in making them available.
>>
>> Here is the BenToWeb site for more information:
>>  - http://www.bentoweb.org/
>>
>> Best,
>>   Shadi
>>
>>
>> On 05-Apr-17 10:59, Wilco Fiers wrote:
>>> Hi everyone,
>>>
>>> After Monday's discussion I had some ideas on how to proceed with our
>>> test
>>> case repository. We've put together a decent size collection of test
>>> cases
>>> at this point. But right now they can't be processed automatically
>>> yet. So
>>> the question is how do we get to that point.
>>>
>>> It seems to me that it would be best if we could let each of these test
>>> case repos be what they are today. I think we should probably avoid
>>> forking
>>> them all and putting them into a new place. That will be way too much
>>> effort to maintain. So what we could potentially do instead, is that for
>>> each of these we build a metadata file, that follows a predefined
>>> structure, so that ATTs can run against those metadata files and run the
>>> test cases that way. Here's what I'm thinking (written in JSON):
>>>
>>> {
>>>   "name": "axe-core",
>>>   "version": "2.1.7",
>>>   "license": "MPL-2.0",
>>>   "a11y-testcases": [{
>>>     "url": "
>>> https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html
>>>
>>>
>>> ",
>>>     "successCriteria": ["wcag20:text-equiv-all"]
>>>     "failed": ["#violation1", "#violation2", "#violation3",
>>> "#violation4"],
>>>     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
>>> "#pass6"]
>>>   }, {
>>>     ...
>>>   }]
>>> }
>>>
>>> Thoughts?
>>>
>>> W
>>>
>>> On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com>
>>> wrote:
>>>
>>>> Sorry Alistair for not replying. I missed that part of the email. Upon
>>>> first reading it I went and added in some links into the wiki page and
>>>> didn't send an email figuring you were following the page and would get
>>>> notifications of updates. Obviously, we have access to IBM rules that
>>>> are
>>>> not open-source and will share knowledge, if not directly the test
>>>> cases so
>>>> did not add them to the list. I am searching the internet for other
>>>> potential candidates and will add what I find there as well.
>>>>
>>>> As for the other email regarding the test case template to follow, I
>>>> like
>>>> the 5 parts of the template (copied below so folks don't have to link
>>>> over
>>>> to your other email).
>>>>
>>>>    1)       Code snippet – element; group of elements; full html
>>>> element
>>>>    contents;
>>>>    2)       Provided by (organisation)
>>>>    3)       From the test suite of which tool
>>>>    4)       Expected outcome – is conforming content; is not conforming
>>>>    content (number of expected issues)
>>>>    5)       Related WCAG Success Criteria
>>>>
>>>>
>>>> Wondering if we would need the following in the template:
>>>>
>>>>    - Title or short name for the test case
>>>>    - an optional "Notes" section where any known accessibility support
>>>>    limitations, or any other explanations of the test case would be
>>>> documented.
>>>>    - Related WCAG implementation technique (if it is an example of one)
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Mary Jo
>>>> ------------------------------
>>>> <http://www.ibm.com/able>
>>>> *Mary Jo Mueller*
>>>> Accessibility Standards Program Manager
>>>> IBM Accessibility, IBM Research, Austin, TX
>>>> Phone: 512-286-9698 <(512)%20286-9698> | Tie-line: 363-9698
>>>> *Search for accessibility answers* <http://ibm.biz/a11y-search>
>>>> <http://www.linkedin.com/e/vgh/2419815/eml-grp-sub/>
>>>> <http://www.facebook.com/IBMAccessibility>
>>>> <http://twitter.com/IBMAccess>
>>>> "If your actions inspire others to dream more, learn more, do more and
>>>> become more, you are a leader."
>>>> *~John Quincy Adams*
>>>>
>>>> [image: Inactive hide details for Alistair Garrison ---03/22/2017
>>>> 10:35:45
>>>> AM---Dear all, I’m in the process of adding a section on “]Alistair
>>>> Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in the process of
>>>> adding
>>>> a section on “Test Case Repositories for WCAG 2.0” to the Tes
>>>>
>>>> From: Alistair Garrison <alistair.garrison@ssbbartgroup.com>
>>>> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
>>>> Date: 03/22/2017 10:35 AM
>>>> Subject: Pooling test case repositories on the wiki
>>>> ------------------------------
>>>>
>>>>
>>>>
>>>> Dear all,
>>>>
>>>> I’m in the process of adding a section on “Test Case Repositories for
>>>> WCAG
>>>> 2.0” to the Testing Resources Wiki (
>>>> *https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources*
>>>>
>>>>
>>>> <https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources>
>>>>
>>>>
>>>> ).
>>>>
>>>> The repositories to be listed in this section will contain examples of
>>>> what constitutes conformance / non-conformance with WCAG 2.0 SCs, in
>>>> terms
>>>> of concrete HTML snippet examples.
>>>>
>>>> For example:
>>>>
>>>>    1)
>>>>
>>>> *https://github.com/dequelabs/axe-core/tree/master/test/integration/full*
>>>>
>>>>
>>>>
>>>> <https://github.com/dequelabs/axe-core/tree/master/test/integration/full>
>>>>
>>>>
>>>>       2) *https://github.com/IBMa/Va11yS*
>>>> <https://github.com/IBMa/Va11yS>
>>>>
>>>>
>>>> Please email me if you have access to, or know of, any further
>>>> publically
>>>> available Test Case Repositories? The more repos the merrier!
>>>>
>>>> FYI – Here is a link from my previous email relating to Test Case
>>>> Repositories -
>>>> *http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html*
>>>> <http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html>
>>>>
>>>> Very best regards
>>>>
>>>> Alistair
>>>>
>>>> ---
>>>>
>>>> Alistair Garrison
>>>> Director of Accessibility Research
>>>> Level-Access (formerly SSB Bart Group)
>>>> Visit us online: *Website* <http://www.ssbbartgroup.com/> | *Twitter*
>>>> <https://twitter.com/SSBBARTGroup> | *Facebook*
>>>> <https://www.facebook.com/ssbbartgroup> | *LinkedIn*
>>>> <https://www.linkedin.com/company/355266?trk=tyah> | *Blog*
>>>> <http://www.ssbbartgroup.com/blog/>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)

Received on Tuesday, 11 April 2017 10:07:54 UTC