Re: Test case format (explicit results)

Hi,


On 15/12/2017 15:19, Alistair Garrison wrote:
> I’m interested in altering the Test case format (explicit results) 
> model.  We have the following on the table at present:
> 
> {
> 
>    "name": "axe-core",
> 
>    "version": "2.1.7",
> 
>    "license": "MPL-2.0",
> 
>    "a11y-testcases": [{
> 
>      "url": 
> "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
> 
>      "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html", 
> "mytool:image-alt"],
> 
>      "failed": ["#violation1", "#violation2", "#violation3",
> 
> "#violation4"],
> 
>      "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
> 
> "#pass6"]
> 
>    }, {
> 
>      ...
> 
>    }]
> 
> }
> 
> I’d be more interested in each piece of “failing or passing” content 
> having it’s own meta – rather than using arrays of bunched data.  The 
> reason being that adding an example should not break loads of other 
> stuff, our cause a maintenance headache.

Indeed, during the development of EARL we had looked at aggregation and 
encountered many problems. The statement above says that a web page has 
4 violations and 6 passes to three tests, but the mappings are unclear. 
The statement is difficult to process any further beyond this point.


> We’re also looking to automate the transpile of our unit test content 
> into this format, and that is aided by having individual test cases 
> defined separately.
> 
> I would propose the following:
> 
> {
> 
>    "organisation": "",
> 
>    "license": "",
> 
>    "a11y-testcases": [
> 
> {
> 
>      "url" or "htmlFragment": 
> "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html#monkeys",
> 
>      "wcagRefs": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
> 
>      "conforms":false | true]
> 
> },
> 
> {
> 
>    etc… per piece of content...

This maps pretty nearly to EARL:

- organisation -> earl:Assertor
- url/htmlFragment -> earl:TestSubject
- wcagRefs -> earl:TestCriterion
- conforms -> TestResult


> The option to provide a url *or HTML fragment* would be very useful in 
> terms of consuming other peoples data…

An earl:TestSubject could be any of:

- simple URI
- "content" (simple text, Base64, or XML)
- HTTP response (headers, and/or body)
- "Document" (description of document)
- "Software" (description of software)


> The “conforms” status is simply set to a true or false.

An earl:TestResult consists of:

- outcome (pass, fail, cannot tell, not tested, not applicable)
- "info" (optional - simple text, for additional information)
- "pointer" (optional - to identify location within TestSubject)


> I have also dropped the testId as it’s not necessary, and could make the 
> test cases subject to change.
> 
> Thoughts / comments welcome.  But, it would be good to get this 
> nailed-down as soon as possible, so we can move ahead with publishing 
> the test cases…

I'm wondering if we should attempt an updated Working Group Note of 
EARL, only this time using JSON rather than RDF/XML serialization for 
the examples? It looks to me like everything is pretty much there.
  - https://www.w3.org/TR/EARL10-Schema/

Best,
   Shadi


> Alistair
> 

-- 
Shadi Abou-Zahra - http://www.w3.org/People/shadi/
Accessibility Strategy and Technology Specialist
Web Accessibility Initiative (WAI)
World Wide Web Consortium (W3C)

Received on Monday, 18 December 2017 06:58:20 UTC