Re: Test case format (explicit results)

Hi Wilco,

We don’t use ids in our own test content – as we use HTML Fragments, within Jasmine unit tests.  The reasons we avoid ids being that:


·         I don’t like the use of ids to identify test content, as ids can of course form part of the material / mechanisms being tested [I was illustrating with your aXe unit test, which do seemingly use ids…  so not using ids is fine with me…]


·         I also don’t like the clustering of multiple pieces of test content into a single test page – as bleed is always possible between pieces of content;


·         I also don’t like “passed” or “failed” as fields, as I’ve already noted – as this is merely there I suspect to support using blocks of test content identified with ids written into a single page…

Can you point to an example of Va11ys content, or Bentoweb content so we can see how they are presented?

We don’t of course need to agree the format that our unit test content is published in – as it is the act of publishing them publically that is the main thing.  I suspect, however, that we’d simply opt for using Shadi’s suggestion, as it seems to closely map to what we do already… It also is in line with the previous EARL work, which I believe this group might “do something with” going forward…

Alistair

From: Wilco Fiers <wilco.fiers@deque.com>
Date: Monday, 18 December 2017 at 10:08
To: Shadi Abou-Zahra <shadi@w3.org>
Cc: Alistair Garrison <alistair.garrison@levelaccess.com>, WCAG ACT TF <public-wcag-act@w3.org>
Subject: Re: Test case format (explicit results)

Hi Alistair, Shadi,

If you output your data like this, you can have them as single items. There is nothing that prevents you from having the URL multiple times. It just create a bunch of duplication.

{
  "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
  "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
  "passed": ["#pass1"]
}, {
  "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
  "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
  "passed": ["#pass2"]
}, ...

As for your proposal. I don't think this works for Va11ys, or for the Bentoweb project. Neither of those uses IDs to point to their results. The Bentoweb examples use XPath (which is easily transformed to CSS), and IBM doesn't have selectors for Va11ys. One of the goals for this was to allow it to work for existing libraries. What you're proposing doesn't seem to do that, where as the existing proposal does allow your use case.

 W

On Mon, Dec 18, 2017 at 7:58 AM, Shadi Abou-Zahra <shadi@w3.org<mailto:shadi@w3.org>> wrote:
Hi,


On 15/12/2017 15:19, Alistair Garrison wrote:
I’m interested in altering the Test case format (explicit results) model.  We have the following on the table at present:

{

   "name": "axe-core",

   "version": "2.1.7",

   "license": "MPL-2.0",

   "a11y-testcases": [{

     "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",

     "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html", "mytool:image-alt"],

     "failed": ["#violation1", "#violation2", "#violation3",

"#violation4"],

     "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",

"#pass6"]

   }, {

     ...

   }]

}

I’d be more interested in each piece of “failing or passing” content having it’s own meta – rather than using arrays of bunched data.  The reason being that adding an example should not break loads of other stuff, our cause a maintenance headache.

Indeed, during the development of EARL we had looked at aggregation and encountered many problems. The statement above says that a web page has 4 violations and 6 passes to three tests, but the mappings are unclear. The statement is difficult to process any further beyond this point.

We’re also looking to automate the transpile of our unit test content into this format, and that is aided by having individual test cases defined separately.

I would propose the following:

{

   "organisation": "",

   "license": "",

   "a11y-testcases": [

{

     "url" or "htmlFragment": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html#monkeys",

     "wcagRefs": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],

     "conforms":false | true]

},

{

   etc… per piece of content...

This maps pretty nearly to EARL:

- organisation -> earl:Assertor
- url/htmlFragment -> earl:TestSubject
- wcagRefs -> earl:TestCriterion
- conforms -> TestResult

The option to provide a url *or HTML fragment* would be very useful in terms of consuming other peoples data…

An earl:TestSubject could be any of:

- simple URI
- "content" (simple text, Base64, or XML)
- HTTP response (headers, and/or body)
- "Document" (description of document)
- "Software" (description of software)

The “conforms” status is simply set to a true or false.

An earl:TestResult consists of:

- outcome (pass, fail, cannot tell, not tested, not applicable)
- "info" (optional - simple text, for additional information)
- "pointer" (optional - to identify location within TestSubject)

I have also dropped the testId as it’s not necessary, and could make the test cases subject to change.

Thoughts / comments welcome.  But, it would be good to get this nailed-down as soon as possible, so we can move ahead with publishing the test cases…

I'm wondering if we should attempt an updated Working Group Note of EARL, only this time using JSON rather than RDF/XML serialization for the examples? It looks to me like everything is pretty much there.
 - https://www.w3.org/TR/EARL10-Schema/


Best,
  Shadi

Alistair

--
Shadi Abou-Zahra - http://www.w3.org/People/shadi/

Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)



--
Wilco Fiers
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
[cid:image001.gif@01D377EC.046BB0B0]