Re: Pooling test case repositories on the wiki

Hi all,

The full set of test cases is available (in 3 versions) in GitHub 
(Apache license):

- https://github.com/webcc/bentoweb-wcag20-test-suite-v1
- https://github.com/webcc/bentoweb-wcag20-test-suite-v2
- https://github.com/webcc/bentoweb-wcag20-test-suite-v3

See also:

<http://bentoweb.org/tools.html>

On 10/04/17 17:07, Shadi Abou-Zahra wrote:
> Hi all,
>
> In 2006 the (now closed) Evaluation and Repair Tools Working Group (ERT
> WG) worked on a "WCAG 2.0 Test Samples Repository":
>  - https://www.w3.org/WAI/ER/tests/
>
> These were contributed by the EC-funded BenToWeb project, which also
> developed a comprehensive Test Case Description Language (TCDL). These
> Test Samples use a sub-set of the TCDL metadata (documented at the URL
> above). We never fully migrated all the Test Samples from the project,
> but I'm sure Fraunhofer would be interested in making them available.
>
> Here is the BenToWeb site for more information:
>  - http://www.bentoweb.org/
>
> Best,
>   Shadi
>
>
> On 05-Apr-17 10:59, Wilco Fiers wrote:
>> Hi everyone,
>>
>> After Monday's discussion I had some ideas on how to proceed with our
>> test
>> case repository. We've put together a decent size collection of test
>> cases
>> at this point. But right now they can't be processed automatically
>> yet. So
>> the question is how do we get to that point.
>>
>> It seems to me that it would be best if we could let each of these test
>> case repos be what they are today. I think we should probably avoid
>> forking
>> them all and putting them into a new place. That will be way too much
>> effort to maintain. So what we could potentially do instead, is that for
>> each of these we build a metadata file, that follows a predefined
>> structure, so that ATTs can run against those metadata files and run the
>> test cases that way. Here's what I'm thinking (written in JSON):
>>
>> {
>>   "name": "axe-core",
>>   "version": "2.1.7",
>>   "license": "MPL-2.0",
>>   "a11y-testcases": [{
>>     "url": "
>> https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html
>>
>> ",
>>     "successCriteria": ["wcag20:text-equiv-all"]
>>     "failed": ["#violation1", "#violation2", "#violation3",
>> "#violation4"],
>>     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
>> "#pass6"]
>>   }, {
>>     ...
>>   }]
>> }
>>
>> Thoughts?
>>
>> W
>>
>> On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com>
>> wrote:
>>
>>> Sorry Alistair for not replying. I missed that part of the email. Upon
>>> first reading it I went and added in some links into the wiki page and
>>> didn't send an email figuring you were following the page and would get
>>> notifications of updates. Obviously, we have access to IBM rules that
>>> are
>>> not open-source and will share knowledge, if not directly the test
>>> cases so
>>> did not add them to the list. I am searching the internet for other
>>> potential candidates and will add what I find there as well.
>>>
>>> As for the other email regarding the test case template to follow, I
>>> like
>>> the 5 parts of the template (copied below so folks don't have to link
>>> over
>>> to your other email).
>>>
>>>    1)       Code snippet – element; group of elements; full html element
>>>    contents;
>>>    2)       Provided by (organisation)
>>>    3)       From the test suite of which tool
>>>    4)       Expected outcome – is conforming content; is not conforming
>>>    content (number of expected issues)
>>>    5)       Related WCAG Success Criteria
>>>
>>>
>>> Wondering if we would need the following in the template:
>>>
>>>    - Title or short name for the test case
>>>    - an optional "Notes" section where any known accessibility support
>>>    limitations, or any other explanations of the test case would be
>>> documented.
>>>    - Related WCAG implementation technique (if it is an example of one)
>>>
>>>
>>> Best regards,
>>>
>>> Mary Jo
>>> ------------------------------
>>> <http://www.ibm.com/able>
>>> *Mary Jo Mueller*
>>> Accessibility Standards Program Manager
>>> IBM Accessibility, IBM Research, Austin, TX
>>> Phone: 512-286-9698 <(512)%20286-9698> | Tie-line: 363-9698
>>> *Search for accessibility answers* <http://ibm.biz/a11y-search>
>>> <http://www.linkedin.com/e/vgh/2419815/eml-grp-sub/>
>>> <http://www.facebook.com/IBMAccessibility>
>>> <http://twitter.com/IBMAccess>
>>> "If your actions inspire others to dream more, learn more, do more and
>>> become more, you are a leader."
>>> *~John Quincy Adams*
>>>
>>> [image: Inactive hide details for Alistair Garrison ---03/22/2017
>>> 10:35:45
>>> AM---Dear all, I’m in the process of adding a section on “]Alistair
>>> Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in the process of
>>> adding
>>> a section on “Test Case Repositories for WCAG 2.0” to the Tes
>>>
>>> From: Alistair Garrison <alistair.garrison@ssbbartgroup.com>
>>> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
>>> Date: 03/22/2017 10:35 AM
>>> Subject: Pooling test case repositories on the wiki
>>> ------------------------------
>>>
>>>
>>>
>>> Dear all,
>>>
>>> I’m in the process of adding a section on “Test Case Repositories for
>>> WCAG
>>> 2.0” to the Testing Resources Wiki (
>>> *https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources*
>>>
>>> <https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources>
>>>
>>> ).
>>>
>>> The repositories to be listed in this section will contain examples of
>>> what constitutes conformance / non-conformance with WCAG 2.0 SCs, in
>>> terms
>>> of concrete HTML snippet examples.
>>>
>>> For example:
>>>
>>>    1)
>>>
>>> *https://github.com/dequelabs/axe-core/tree/master/test/integration/full*
>>>
>>>
>>> <https://github.com/dequelabs/axe-core/tree/master/test/integration/full>
>>>
>>>       2) *https://github.com/IBMa/Va11yS*
>>> <https://github.com/IBMa/Va11yS>
>>>
>>>
>>> Please email me if you have access to, or know of, any further
>>> publically
>>> available Test Case Repositories? The more repos the merrier!
>>>
>>> FYI – Here is a link from my previous email relating to Test Case
>>> Repositories -
>>> *http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html*
>>> <http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html>
>>>
>>> Very best regards
>>>
>>> Alistair
>>>
>>> ---
>>>
>>> Alistair Garrison
>>> Director of Accessibility Research
>>> Level-Access (formerly SSB Bart Group)
>>> Visit us online: *Website* <http://www.ssbbartgroup.com/> | *Twitter*
>>> <https://twitter.com/SSBBARTGroup> | *Facebook*
>>> <https://www.facebook.com/ssbbartgroup> | *LinkedIn*
>>> <https://www.linkedin.com/company/355266?trk=tyah> | *Blog*
>>> <http://www.ssbbartgroup.com/blog/>
>>>
>>>
>>>
>>>
>>>
>>
>>
>

-- 
Best Regards, Mit freundlichen Grüßen, Saludos,
carlos

Dr Carlos A Velasco
   Web Compliance Center
   Fraunhofer Institute for Applied Information Technology FIT
   Schloss Birlinghoven, D53757 Sankt Augustin (Germany)
   https://imergo.com/ · http://www.fit.fraunhofer.de/
   Tel: +49-2241-142609 · Fax: +49-2241-1442609

Received on Tuesday, 11 April 2017 09:25:32 UTC