Re: Pooling test case repositories on the wiki

Hi all,

In 2006 the (now closed) Evaluation and Repair Tools Working Group (ERT 
WG) worked on a "WCAG 2.0 Test Samples Repository":
  - https://www.w3.org/WAI/ER/tests/

These were contributed by the EC-funded BenToWeb project, which also 
developed a comprehensive Test Case Description Language (TCDL). These 
Test Samples use a sub-set of the TCDL metadata (documented at the URL 
above). We never fully migrated all the Test Samples from the project, 
but I'm sure Fraunhofer would be interested in making them available.

Here is the BenToWeb site for more information:
  - http://www.bentoweb.org/

Best,
   Shadi


On 05-Apr-17 10:59, Wilco Fiers wrote:
> Hi everyone,
>
> After Monday's discussion I had some ideas on how to proceed with our test
> case repository. We've put together a decent size collection of test cases
> at this point. But right now they can't be processed automatically yet. So
> the question is how do we get to that point.
>
> It seems to me that it would be best if we could let each of these test
> case repos be what they are today. I think we should probably avoid forking
> them all and putting them into a new place. That will be way too much
> effort to maintain. So what we could potentially do instead, is that for
> each of these we build a metadata file, that follows a predefined
> structure, so that ATTs can run against those metadata files and run the
> test cases that way. Here's what I'm thinking (written in JSON):
>
> {
>   "name": "axe-core",
>   "version": "2.1.7",
>   "license": "MPL-2.0",
>   "a11y-testcases": [{
>     "url": "
> https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html
> ",
>     "successCriteria": ["wcag20:text-equiv-all"]
>     "failed": ["#violation1", "#violation2", "#violation3", "#violation4"],
>     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5", "#pass6"]
>   }, {
>     ...
>   }]
> }
>
> Thoughts?
>
> W
>
> On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com>
> wrote:
>
>> Sorry Alistair for not replying. I missed that part of the email. Upon
>> first reading it I went and added in some links into the wiki page and
>> didn't send an email figuring you were following the page and would get
>> notifications of updates. Obviously, we have access to IBM rules that are
>> not open-source and will share knowledge, if not directly the test cases so
>> did not add them to the list. I am searching the internet for other
>> potential candidates and will add what I find there as well.
>>
>> As for the other email regarding the test case template to follow, I like
>> the 5 parts of the template (copied below so folks don't have to link over
>> to your other email).
>>
>>    1)       Code snippet – element; group of elements; full html element
>>    contents;
>>    2)       Provided by (organisation)
>>    3)       From the test suite of which tool
>>    4)       Expected outcome – is conforming content; is not conforming
>>    content (number of expected issues)
>>    5)       Related WCAG Success Criteria
>>
>>
>> Wondering if we would need the following in the template:
>>
>>    - Title or short name for the test case
>>    - an optional "Notes" section where any known accessibility support
>>    limitations, or any other explanations of the test case would be documented.
>>    - Related WCAG implementation technique (if it is an example of one)
>>
>>
>> Best regards,
>>
>> Mary Jo
>> ------------------------------
>> <http://www.ibm.com/able>
>> *Mary Jo Mueller*
>> Accessibility Standards Program Manager
>> IBM Accessibility, IBM Research, Austin, TX
>> Phone: 512-286-9698 <(512)%20286-9698> | Tie-line: 363-9698
>> *Search for accessibility answers* <http://ibm.biz/a11y-search>
>> <http://www.linkedin.com/e/vgh/2419815/eml-grp-sub/>
>> <http://www.facebook.com/IBMAccessibility> <http://twitter.com/IBMAccess>
>> "If your actions inspire others to dream more, learn more, do more and
>> become more, you are a leader."
>> *~John Quincy Adams*
>>
>> [image: Inactive hide details for Alistair Garrison ---03/22/2017 10:35:45
>> AM---Dear all, I’m in the process of adding a section on “]Alistair
>> Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in the process of adding
>> a section on “Test Case Repositories for WCAG 2.0” to the Tes
>>
>> From: Alistair Garrison <alistair.garrison@ssbbartgroup.com>
>> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
>> Date: 03/22/2017 10:35 AM
>> Subject: Pooling test case repositories on the wiki
>> ------------------------------
>>
>>
>>
>> Dear all,
>>
>> I’m in the process of adding a section on “Test Case Repositories for WCAG
>> 2.0” to the Testing Resources Wiki (
>> *https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources*
>> <https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources>
>> ).
>>
>> The repositories to be listed in this section will contain examples of
>> what constitutes conformance / non-conformance with WCAG 2.0 SCs, in terms
>> of concrete HTML snippet examples.
>>
>> For example:
>>
>>    1)
>>       *https://github.com/dequelabs/axe-core/tree/master/test/integration/full*
>>       <https://github.com/dequelabs/axe-core/tree/master/test/integration/full>
>>       2) *https://github.com/IBMa/Va11yS* <https://github.com/IBMa/Va11yS>
>>
>>
>> Please email me if you have access to, or know of, any further publically
>> available Test Case Repositories? The more repos the merrier!
>>
>> FYI – Here is a link from my previous email relating to Test Case
>> Repositories -
>> *http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html*
>> <http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html>
>>
>> Very best regards
>>
>> Alistair
>>
>> ---
>>
>> Alistair Garrison
>> Director of Accessibility Research
>> Level-Access (formerly SSB Bart Group)
>> Visit us online: *Website* <http://www.ssbbartgroup.com/> | *Twitter*
>> <https://twitter.com/SSBBARTGroup> | *Facebook*
>> <https://www.facebook.com/ssbbartgroup> | *LinkedIn*
>> <https://www.linkedin.com/company/355266?trk=tyah> | *Blog*
>> <http://www.ssbbartgroup.com/blog/>
>>
>>
>>
>>
>>
>
>

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)

Received on Monday, 10 April 2017 15:08:24 UTC