Re: Pooling test case repositories on the wiki

Dear All,

Generally, I like the concept that Wilco proposes.  The change I’d recommend would be to use custom data attributes (e.g. data-expectedoutcome) in the source code under test instead of id attributes; as some tests may relate to id attribute values, or the lack or presence of id attributes.

<img src="img.jpg" data-expectedoutcome="violation">

I’d remove the numbers from violation as above; then change the JSON file to:

“failed”: 4,
“passed”: 6,

It’s a little easier to maintain, and a little less prone to human error creeping in.

Thoughts / comments?

All the best

Alistair

From: Wilco Fiers <wilco.fiers@deque.com>
Date: Wednesday, 5 April 2017 at 09:59
To: Mary Jo Mueller <maryjom@us.ibm.com>
Cc: Alistair Garrison <alistair.garrison@ssbbartgroup.com>, Accessibility Conformance Testing <public-wcag-act@w3.org>
Subject: Re: Pooling test case repositories on the wiki

Hi everyone,

After Monday's discussion I had some ideas on how to proceed with our test case repository. We've put together a decent size collection of test cases at this point. But right now they can't be processed automatically yet. So the question is how do we get to that point.

It seems to me that it would be best if we could let each of these test case repos be what they are today. I think we should probably avoid forking them all and putting them into a new place. That will be way too much effort to maintain. So what we could potentially do instead, is that for each of these we build a metadata file, that follows a predefined structure, so that ATTs can run against those metadata files and run the test cases that way. Here's what I'm thinking (written in JSON):

{
  "name": "axe-core",
  "version": "2.1.7",
  "license": "MPL-2.0",
  "a11y-testcases": [{
    "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
    "successCriteria": ["wcag20:text-equiv-all"]
    "failed": ["#violation1", "#violation2", "#violation3", "#violation4"],
    "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5", "#pass6"]
  }, {
    ...
  }]
}

Thoughts?

W

On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com<mailto:maryjom@us.ibm.com>> wrote:

Sorry Alistair for not replying. I missed that part of the email. Upon first reading it I went and added in some links into the wiki page and didn't send an email figuring you were following the page and would get notifications of updates. Obviously, we have access to IBM rules that are not open-source and will share knowledge, if not directly the test cases so did not add them to the list. I am searching the internet for other potential candidates and will add what I find there as well.

As for the other email regarding the test case template to follow, I like the 5 parts of the template (copied below so folks don't have to link over to your other email).
1)       Code snippet – element; group of elements; full html element contents;
2)       Provided by (organisation)
3)       From the test suite of which tool
4)       Expected outcome – is conforming content; is not conforming content (number of expected issues)
5)       Related WCAG Success Criteria

Wondering if we would need the following in the template:

  *   Title or short name for the test case
  *   an optional "Notes" section where any known accessibility support limitations, or any other explanations of the test case would be documented.
  *   Related WCAG implementation technique (if it is an example of one)

Best regards,

Mary Jo

________________________________

[cid:image001.jpg@01D2B84A.64228400]<http://www.ibm.com/able>


Mary Jo Mueller
Accessibility Standards Program Manager
IBM Accessibility, IBM Research, Austin, TX
Phone: 512-286-9698<tel:(512)%20286-9698> | Tie-line: 363-9698
Search for accessibility answers<http://ibm.biz/a11y-search> [cid:image002.jpg@01D2B84A.64228400]



[cid:image004.gif@01D2B84A.64228400]




















"If your actions inspire others to dream more, learn more, do more and become more, you are a leader."
~John Quincy Adams

[nactive hide details for Alistair Garrison ---03/22/2017 10:35:45 AM---De]Alistair Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in the process of adding a section on “Test Case Repositories for WCAG 2.0” to the Tes

From: Alistair Garrison <alistair.garrison@ssbbartgroup.com<mailto:alistair.garrison@ssbbartgroup.com>>
To: Accessibility Conformance Testing <public-wcag-act@w3.org<mailto:public-wcag-act@w3.org>>
Date: 03/22/2017 10:35 AM
Subject: Pooling test case repositories on the wiki
________________________________



Dear all,

I’m in the process of adding a section on “Test Case Repositories for WCAG 2.0” to the Testing Resources Wiki (https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources).

The repositories to be listed in this section will contain examples of what constitutes conformance / non-conformance with WCAG 2.0 SCs, in terms of concrete HTML snippet examples.

For example:
1) https://github.com/dequelabs/axe-core/tree/master/test/integration/full

2) https://github.com/IBMa/Va11yS


Please email me if you have access to, or know of, any further publically available Test Case Repositories? The more repos the merrier!

FYI – Here is a link from my previous email relating to Test Case Repositories - http://lists.w3.org/Archives/Public/public-wcag-act/2017Feb/0007.html


Very best regards

Alistair

---

Alistair Garrison
Director of Accessibility Research
Level-Access (formerly SSB Bart Group)
Visit us online: Website<http://www.ssbbartgroup.com/> | Twitter<https://twitter.com/SSBBARTGroup> | Facebook<https://www.facebook.com/ssbbartgroup> | LinkedIn<https://www.linkedin.com/company/355266?trk=tyah> | Blog<http://www.ssbbartgroup.com/blog/>






--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:image015.gif@01D2B84A.64228400]