Re: Test case format (explicit results)

Hi Alistair, Shadi,

If you output your data like this, you can have them as single items. There
is nothing that prevents you from having the URL multiple times. It just
create a bunch of duplication.

{
  "url": "
https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html
",
  "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
  "passed": ["#pass1"]
}, {
  "url": "
https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html
",
  "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
  "passed": ["#pass2"]
}, ...

As for your proposal. I don't think this works for Va11ys, or for the
Bentoweb project. Neither of those uses IDs to point to their results. The
Bentoweb examples use XPath (which is easily transformed to CSS), and IBM
doesn't have selectors for Va11ys. One of the goals for this was to allow
it to work for existing libraries. What you're proposing doesn't seem to do
that, where as the existing proposal does allow your use case.

 W

On Mon, Dec 18, 2017 at 7:58 AM, Shadi Abou-Zahra <shadi@w3.org> wrote:

> Hi,
>
>
> On 15/12/2017 15:19, Alistair Garrison wrote:
>
>> I’m interested in altering the Test case format (explicit results)
>> model.  We have the following on the table at present:
>>
>> {
>>
>>    "name": "axe-core",
>>
>>    "version": "2.1.7",
>>
>>    "license": "MPL-2.0",
>>
>>    "a11y-testcases": [{
>>
>>      "url": "https://raw.githubusercontent.com/dequelabs/axe-core/
>> master/test/integration/rules/image-alt/image-alt.html",
>>
>>      "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html",
>> "mytool:image-alt"],
>>
>>      "failed": ["#violation1", "#violation2", "#violation3",
>>
>> "#violation4"],
>>
>>      "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
>>
>> "#pass6"]
>>
>>    }, {
>>
>>      ...
>>
>>    }]
>>
>> }
>>
>> I’d be more interested in each piece of “failing or passing” content
>> having it’s own meta – rather than using arrays of bunched data.  The
>> reason being that adding an example should not break loads of other stuff,
>> our cause a maintenance headache.
>>
>
> Indeed, during the development of EARL we had looked at aggregation and
> encountered many problems. The statement above says that a web page has 4
> violations and 6 passes to three tests, but the mappings are unclear. The
> statement is difficult to process any further beyond this point.
>
>
> We’re also looking to automate the transpile of our unit test content into
>> this format, and that is aided by having individual test cases defined
>> separately.
>>
>> I would propose the following:
>>
>> {
>>
>>    "organisation": "",
>>
>>    "license": "",
>>
>>    "a11y-testcases": [
>>
>> {
>>
>>      "url" or "htmlFragment": "https://raw.githubusercontent
>> .com/dequelabs/axe-core/master/test/integration/rules/
>> image-alt/image-alt.html#monkeys",
>>
>>      "wcagRefs": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>>
>>      "conforms":false | true]
>>
>> },
>>
>> {
>>
>>    etc… per piece of content...
>>
>
> This maps pretty nearly to EARL:
>
> - organisation -> earl:Assertor
> - url/htmlFragment -> earl:TestSubject
> - wcagRefs -> earl:TestCriterion
> - conforms -> TestResult
>
>
> The option to provide a url *or HTML fragment* would be very useful in
>> terms of consuming other peoples data…
>>
>
> An earl:TestSubject could be any of:
>
> - simple URI
> - "content" (simple text, Base64, or XML)
> - HTTP response (headers, and/or body)
> - "Document" (description of document)
> - "Software" (description of software)
>
>
> The “conforms” status is simply set to a true or false.
>>
>
> An earl:TestResult consists of:
>
> - outcome (pass, fail, cannot tell, not tested, not applicable)
> - "info" (optional - simple text, for additional information)
> - "pointer" (optional - to identify location within TestSubject)
>
>
> I have also dropped the testId as it’s not necessary, and could make the
>> test cases subject to change.
>>
>> Thoughts / comments welcome.  But, it would be good to get this
>> nailed-down as soon as possible, so we can move ahead with publishing the
>> test cases…
>>
>
> I'm wondering if we should attempt an updated Working Group Note of EARL,
> only this time using JSON rather than RDF/XML serialization for the
> examples? It looks to me like everything is pretty much there.
>  - https://www.w3.org/TR/EARL10-Schema/
>
> Best,
>   Shadi
>
>
> Alistair
>>
>>
> --
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> Accessibility Strategy and Technology Specialist
> Web Accessibility Initiative (WAI)
> World Wide Web Consortium (W3C)
>
>


-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG