Re: Test case format (explicit results)

Hi Wilco,

I’m sure you’ll elaborate in the meeting this week, but can I just ask what exactly about the format is “looking to be a lot more complicated than I [you] initially expected it was going to be” – so we can all have a group think before Thurs?

Personally, I think we need this for “9.1 Implementation Validation”? And, personally I don’t think we need to wait for EARL 2 – Shadi’s EARL 1 suggestion looked perfect (simple + easy to implement right now) to me…

All the best

Alistair

From: Wilco Fiers <wilco.fiers@deque.com>
Date: Monday, 18 December 2017 at 15:58
To: Alistair Garrison <alistair.garrison@levelaccess.com>
Cc: Shadi Abou-Zahra <shadi@w3.org>, WCAG ACT TF <public-wcag-act@w3.org>
Subject: Re: Test case format (explicit results)

Hi Alistair,
I think a metadata format for test cases isn't part of the work statement for this group. It would have been a nice addition, but it's looking to be a lot more complicated then I initially expected it was going to be. We just had a talk about it in the ACT planning call. I've put the topic on the agenda for this week. We may just want to move this work into another group. There has already been talk about revisiting EARL. This might be better suited in that discussion.

Wilco

On Mon, Dec 18, 2017 at 4:49 PM, Alistair Garrison <alistair.garrison@levelaccess.com<mailto:alistair.garrison@levelaccess.com>> wrote:
Hi,

Isn’t the format for presenting test content against a known outcome part of “9.1 Implementation Validation”?

If yes, don’t we have to agree a format in this group?  Shadi’s suggested EARL format, for example, would I think also fit nicely with other pieces of the work – like test results, and the EARL aggregation model used in Appendix 1…

Maybe I’m wrong?

Alistair

From: Wilco Fiers <wilco.fiers@deque.com<mailto:wilco.fiers@deque.com>>
Date: Monday, 18 December 2017 at 15:04

To: Alistair Garrison <alistair.garrison@levelaccess.com<mailto:alistair.garrison@levelaccess.com>>
Cc: Shadi Abou-Zahra <shadi@w3.org<mailto:shadi@w3.org>>, WCAG ACT TF <public-wcag-act@w3.org<mailto:public-wcag-act@w3.org>>
Subject: Re: Test case format (explicit results)

Hi Alistair,
Apologies. I didn't mean to imply this JSON structure was ever formally voted on. It's not a deliverable for ACT TF, it's just something that would be a nice bonus it it could be done in a way that supports the development of the ACT Rules format. We used this to base the test cases for the ACT Rules examples on. Those were voted on before we published them. For instance this one:
https://w3c.github.io/wcag-act-rules/rule-tests/ACT-R2-tests.html


If you want to track the discussion, I looked up some of the minutes for you:
https://www.w3.org/2017/09/25-wcag-act-minutes.html

https://www.w3.org/2017/06/05-wcag-act-minutes.html

https://www.w3.org/2017/05/15-wcag-act-minutes.html

https://www.w3.org/2017/04/24-wcag-act-minutes.html



W

On Mon, Dec 18, 2017 at 3:06 PM, Alistair Garrison <alistair.garrison@levelaccess.com<mailto:alistair.garrison@levelaccess.com>> wrote:
Hi Wilco,

I’m sorry – I can’t remember the ACT-group vote on that format, maybe I was away – I’ll happily check it out.  Can you send me a link to the vote, or minutes containing the vote?

All the best

Alistair

From: Wilco Fiers <wilco.fiers@deque.com<mailto:wilco.fiers@deque.com>>
Date: Monday, 18 December 2017 at 12:54
To: Alistair Garrison <alistair.garrison@levelaccess.com<mailto:alistair.garrison@levelaccess.com>>
Cc: Shadi Abou-Zahra <shadi@w3.org<mailto:shadi@w3.org>>, WCAG ACT TF <public-wcag-act@w3.org<mailto:public-wcag-act@w3.org>>

Subject: Re: Test case format (explicit results)

Hi Alistair,
The format that's there now has been agreed on by participants from Deque, IBM and SiteImprove. This work was planned and finished months ago, and we've already put time into the implementation of it. I think if things need to be added to it, that's fine, but to ask us to redo our implementation work because there are features that you don't need, that's not something I'm keen on unless there are tangible problems with it.

As for the links, we created a list with several test case repositories. They are on the same page as the proposed format: https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources#Accessibility_Testing_Repositories


W

On Mon, Dec 18, 2017 at 11:36 AM, Alistair Garrison <alistair.garrison@levelaccess.com<mailto:alistair.garrison@levelaccess.com>> wrote:
Hi Wilco,

We don’t use ids in our own test content – as we use HTML Fragments, within Jasmine unit tests.  The reasons we avoid ids being that:


•         I don’t like the use of ids to identify test content, as ids can of course form part of the material / mechanisms being tested [I was illustrating with your aXe unit test, which do seemingly use ids…  so not using ids is fine with me…]


•         I also don’t like the clustering of multiple pieces of test content into a single test page – as bleed is always possible between pieces of content;


•         I also don’t like “passed” or “failed” as fields, as I’ve already noted – as this is merely there I suspect to support using blocks of test content identified with ids written into a single page…

Can you point to an example of Va11ys content, or Bentoweb content so we can see how they are presented?

We don’t of course need to agree the format that our unit test content is published in – as it is the act of publishing them publically that is the main thing.  I suspect, however, that we’d simply opt for using Shadi’s suggestion, as it seems to closely map to what we do already… It also is in line with the previous EARL work, which I believe this group might “do something with” going forward…

Alistair

From: Wilco Fiers <wilco.fiers@deque.com<mailto:wilco.fiers@deque.com>>
Date: Monday, 18 December 2017 at 10:08
To: Shadi Abou-Zahra <shadi@w3.org<mailto:shadi@w3.org>>
Cc: Alistair Garrison <alistair.garrison@levelaccess.com<mailto:alistair.garrison@levelaccess.com>>, WCAG ACT TF <public-wcag-act@w3.org<mailto:public-wcag-act@w3.org>>
Subject: Re: Test case format (explicit results)

Hi Alistair, Shadi,

If you output your data like this, you can have them as single items. There is nothing that prevents you from having the URL multiple times. It just create a bunch of duplication.

{
  "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
  "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
  "passed": ["#pass1"]
}, {
  "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
  "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
  "passed": ["#pass2"]
}, ...

As for your proposal. I don't think this works for Va11ys, or for the Bentoweb project. Neither of those uses IDs to point to their results. The Bentoweb examples use XPath (which is easily transformed to CSS), and IBM doesn't have selectors for Va11ys. One of the goals for this was to allow it to work for existing libraries. What you're proposing doesn't seem to do that, where as the existing proposal does allow your use case.

 W

On Mon, Dec 18, 2017 at 7:58 AM, Shadi Abou-Zahra <shadi@w3.org<mailto:shadi@w3.org>> wrote:
Hi,


On 15/12/2017 15:19, Alistair Garrison wrote:
I’m interested in altering the Test case format (explicit results) model.  We have the following on the table at present:

{

   "name": "axe-core",

   "version": "2.1.7",

   "license": "MPL-2.0",

   "a11y-testcases": [{

     "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",

     "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html", "mytool:image-alt"],

     "failed": ["#violation1", "#violation2", "#violation3",

"#violation4"],

     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",

"#pass6"]

   }, {

     ...

   }]

}

I’d be more interested in each piece of “failing or passing” content having it’s own meta – rather than using arrays of bunched data.  The reason being that adding an example should not break loads of other stuff, our cause a maintenance headache.

Indeed, during the development of EARL we had looked at aggregation and encountered many problems. The statement above says that a web page has 4 violations and 6 passes to three tests, but the mappings are unclear. The statement is difficult to process any further beyond this point.
We’re also looking to automate the transpile of our unit test content into this format, and that is aided by having individual test cases defined separately.

I would propose the following:

{

   "organisation": "",

   "license": "",

   "a11y-testcases": [

{

     "url" or "htmlFragment": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html#monkeys",

     "wcagRefs": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],

     "conforms":false | true]

},

{

   etc… per piece of content...

This maps pretty nearly to EARL:

- organisation -> earl:Assertor
- url/htmlFragment -> earl:TestSubject
- wcagRefs -> earl:TestCriterion
- conforms -> TestResult
The option to provide a url *or HTML fragment* would be very useful in terms of consuming other peoples data…

An earl:TestSubject could be any of:

- simple URI
- "content" (simple text, Base64, or XML)
- HTTP response (headers, and/or body)
- "Document" (description of document)
- "Software" (description of software)
The “conforms” status is simply set to a true or false.

An earl:TestResult consists of:

- outcome (pass, fail, cannot tell, not tested, not applicable)
- "info" (optional - simple text, for additional information)
- "pointer" (optional - to identify location within TestSubject)
I have also dropped the testId as it’s not necessary, and could make the test cases subject to change.

Thoughts / comments welcome.  But, it would be good to get this nailed-down as soon as possible, so we can move ahead with publishing the test cases…

I'm wondering if we should attempt an updated Working Group Note of EARL, only this time using JSON rather than RDF/XML serialization for the examples? It looks to me like everything is pretty much there.
 - https://www.w3.org/TR/EARL10-Schema/


Best,
  Shadi
Alistair

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/

Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)



--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:image001.gif@01D37822.55C177C0]



--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:image002.gif@01D37822.55C177C0]



--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:image003.gif@01D37822.55C177C0]



--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:image004.gif@01D37822.55C177C0]