Re: Pooling test case repositories on the wiki

Hi Alistair,
I agree with your argument of maintaining the test cases. I think we should
make this work without requiring any change to the repository that the test
cases live in, which we can with a JSON file. It could even live outside
the repo somewhere, as long as it's kept in sync, which isn't a problem if
the repo is stale.

How about if we allow for both methods? Either you put selectors in the
JSON file indicating the outcome, or you put them in the HTML files. What
we should probably also do is allow the JSON file to specify selectors for
pass and fail results, so it's not hard coded.

{
   "failedSelector": "*[data-expectedoutcome='violation']",
   "passedSelector": "*[data-expectedoutcome='pass']",
}

Wilco

On Tue, Apr 18, 2017 at 2:48 PM, Alistair Garrison <
alistair.garrison@ssbbartgroup.com> wrote:

> Dear All,
>
>
>
> Generally, I like the concept that Wilco proposes.  The change I’d
> recommend would be to use custom data attributes (e.g.
> data-expectedoutcome) in the source code under test instead of id
> attributes; as some tests may relate to id attribute values, or the lack or
> presence of id attributes.
>
>
>
> <img src="img.jpg" data-expectedoutcome="violation">
>
>
>
> I’d remove the numbers from violation as above; then change the JSON file
> to:
>
>
>
> “failed”: 4,
>
> “passed”: 6,
>
>
>
> It’s a little easier to maintain, and a little less prone to human error
> creeping in.
>
>
>
> Thoughts / comments?
>
>
>
> All the best
>
>
>
> Alistair
>
>
>
> *From: *Wilco Fiers <wilco.fiers@deque.com>
> *Date: *Wednesday, 5 April 2017 at 09:59
> *To: *Mary Jo Mueller <maryjom@us.ibm.com>
> *Cc: *Alistair Garrison <alistair.garrison@ssbbartgroup.com>,
> Accessibility Conformance Testing <public-wcag-act@w3.org>
> *Subject: *Re: Pooling test case repositories on the wiki
>
>
>
> Hi everyone,
>
>
>
> After Monday's discussion I had some ideas on how to proceed with our test
> case repository. We've put together a decent size collection of test cases
> at this point. But right now they can't be processed automatically yet. So
> the question is how do we get to that point.
>
>
>
> It seems to me that it would be best if we could let each of these test
> case repos be what they are today. I think we should probably avoid forking
> them all and putting them into a new place. That will be way too much
> effort to maintain. So what we could potentially do instead, is that for
> each of these we build a metadata file, that follows a predefined
> structure, so that ATTs can run against those metadata files and run the
> test cases that way. Here's what I'm thinking (written in JSON):
>
>
>
> {
>
>   "name": "axe-core",
>
>   "version": "2.1.7",
>
>   "license": "MPL-2.0",
>
>   "a11y-testcases": [{
>
>     "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>     "successCriteria": ["wcag20:text-equiv-all"]
>
>     "failed": ["#violation1", "#violation2", "#violation3", "#violation4"],
>
>     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5", "#pass6"]
>
>   }, {
>
>     ...
>
>   }]
>
> }
>
>
>
> Thoughts?
>
>
>
> W
>
>
>
> On Wed, Mar 29, 2017 at 11:50 PM, Mary Jo Mueller <maryjom@us.ibm.com>
> wrote:
>
> Sorry Alistair for not replying. I missed that part of the email. Upon
> first reading it I went and added in some links into the wiki page and
> didn't send an email figuring you were following the page and would get
> notifications of updates. Obviously, we have access to IBM rules that are
> not open-source and will share knowledge, if not directly the test cases so
> did not add them to the list. I am searching the internet for other
> potential candidates and will add what I find there as well.
>
> As for the other email regarding the test case template to follow, I like
> the 5 parts of the template (copied below so folks don't have to link over
> to your other email).
>
> 1)       Code snippet – element; group of elements; full html element
> contents;
> 2)       Provided by (organisation)
> 3)       From the test suite of which tool
> 4)       Expected outcome – is conforming content; is not conforming
> content (number of expected issues)
> 5)       Related WCAG Success Criteria
>
>
> Wondering if we would need the following in the template:
>
>    - Title or short name for the test case
>    - an optional "Notes" section where any known accessibility support
>    limitations, or any other explanations of the test case would be documented.
>    - Related WCAG implementation technique (if it is an example of one)
>
>
> Best regards,
>
> Mary Jo
> ------------------------------
>
> <http://www.ibm.com/able>
>
>
> *Mary Jo Mueller*
> Accessibility Standards Program Manager
> IBM Accessibility, IBM Research, Austin, TX
> Phone: 512-286-9698 <(512)%20286-9698> | Tie-line: 363-9698
> Search for accessibility answers <http://ibm.biz/a11y-search>
>
> "If your actions inspire others to dream more, learn more, do more and
> become more, you are a leader."
> *~John Quincy Adams*
>
> [image: nactive hide details for Alistair Garrison ---03/22/2017 10:35:45
> AM---De]Alistair Garrison ---03/22/2017 10:35:45 AM---Dear all, I’m in
> the process of adding a section on “Test Case Repositories for WCAG 2.0” to
> the Tes
>
> From: Alistair Garrison <alistair.garrison@ssbbartgroup.com>
> To: Accessibility Conformance Testing <public-wcag-act@w3.org>
> Date: 03/22/2017 10:35 AM
> Subject: Pooling test case repositories on the wiki
>
> ------------------------------
>
>
>
>
> Dear all,
>
> I’m in the process of adding a section on “Test Case Repositories for WCAG
> 2.0” to the Testing Resources Wiki (https://www.w3.org/WAI/GL/
> task-forces/conformance-testing/wiki/Testing_Resources).
>
> The repositories to be listed in this section will contain examples of
> what constitutes conformance / non-conformance with WCAG 2.0 SCs, in terms
> of concrete HTML snippet examples.
>
> For example:
>
> 1) https://github.com/dequelabs/axe-core/tree/master/test/integration/full
> 2) https://github.com/IBMa/Va11yS
>
>
> Please email me if you have access to, or know of, any further publically
> available Test Case Repositories? The more repos the merrier!
>
> FYI – Here is a link from my previous email relating to Test Case
> Repositories - http://lists.w3.org/Archives/Public/public-wcag-act/
> 2017Feb/0007.html
>
> Very best regards
>
> Alistair
>
> ---
>
> Alistair Garrison
> Director of Accessibility Research
> Level-Access (formerly SSB Bart Group)
> Visit us online: Website <http://www.ssbbartgroup.com/> | Twitter
> <https://twitter.com/SSBBARTGroup> | Facebook
> <https://www.facebook.com/ssbbartgroup> | LinkedIn
> <https://www.linkedin.com/company/355266?trk=tyah> | Blog
> <http://www.ssbbartgroup.com/blog/>
>
>
>
>
>
>
>
> --
>
> *Wilco Fiers*
>
> Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
>
>


-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG