Re: Test case format (explicit results)

Hi Alistair,
The format that's there now has been agreed on by participants from Deque,
IBM and SiteImprove. This work was planned and finished months ago, and
we've already put time into the implementation of it. I think if things
need to be added to it, that's fine, but to ask us to redo our
implementation work because there are features that you *don't* need,
that's not something I'm keen on unless there are tangible problems with it.

As for the links, we created a list with several test case repositories.
They are on the same page as the proposed format:
https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources#Accessibility_Testing_Repositories

W

On Mon, Dec 18, 2017 at 11:36 AM, Alistair Garrison <
alistair.garrison@levelaccess.com> wrote:

> Hi Wilco,
>
>
>
> We don’t use ids in our own test content – as we use HTML Fragments,
> within Jasmine unit tests.  The reasons we avoid ids being that:
>
>
>
> ·         I don’t like the use of ids to identify test content, as ids
> can of course form part of the material / mechanisms being tested [I was
> illustrating with your aXe unit test, which do seemingly use ids…  so not
> using ids is fine with me…]
>
>
>
> ·         I also don’t like the clustering of multiple pieces of test
> content into a single test page – as bleed is always possible between
> pieces of content;
>
>
>
> ·         I also don’t like “passed” or “failed” as fields, as I’ve
> already noted – as this is merely there I suspect to support using blocks
> of test content identified with ids written into a single page…
>
>
>
> Can you point to an example of Va11ys content, or Bentoweb content so we
> can see how they are presented?
>
>
>
> We don’t of course need to agree the format that our unit test content is
> published in – as it is the act of publishing them publically that is the
> main thing.  I suspect, however, that we’d simply opt for using Shadi’s
> suggestion, as it seems to closely map to what we do already… It also is in
> line with the previous EARL work, which I believe this group might “do
> something with” going forward…
>
>
>
> Alistair
>
>
>
> *From: *Wilco Fiers <wilco.fiers@deque.com>
> *Date: *Monday, 18 December 2017 at 10:08
> *To: *Shadi Abou-Zahra <shadi@w3.org>
> *Cc: *Alistair Garrison <alistair.garrison@levelaccess.com>, WCAG ACT TF <
> public-wcag-act@w3.org>
> *Subject: *Re: Test case format (explicit results)
>
>
>
> Hi Alistair, Shadi,
>
>
>
> If you output your data like this, you can have them as single items.
> There is nothing that prevents you from having the URL multiple times. It
> just create a bunch of duplication.
>
>
>
> {
>
>   "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>   "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>
>   "passed": ["#pass1"]
>
> }, {
>
>   "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>   "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>
>   "passed": ["#pass2"]
>
> }, ...
>
>
>
> As for your proposal. I don't think this works for Va11ys, or for the
> Bentoweb project. Neither of those uses IDs to point to their results. The
> Bentoweb examples use XPath (which is easily transformed to CSS), and IBM
> doesn't have selectors for Va11ys. One of the goals for this was to allow
> it to work for existing libraries. What you're proposing doesn't seem to do
> that, where as the existing proposal does allow your use case.
>
>
>
>  W
>
>
>
> On Mon, Dec 18, 2017 at 7:58 AM, Shadi Abou-Zahra <shadi@w3.org> wrote:
>
> Hi,
>
>
> On 15/12/2017 15:19, Alistair Garrison wrote:
>
> I’m interested in altering the Test case format (explicit results) model.
> We have the following on the table at present:
>
> {
>
>    "name": "axe-core",
>
>    "version": "2.1.7",
>
>    "license": "MPL-2.0",
>
>    "a11y-testcases": [{
>
>      "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>      "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html",
> "mytool:image-alt"],
>
>      "failed": ["#violation1", "#violation2", "#violation3",
>
> "#violation4"],
>
>      "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
>
> "#pass6"]
>
>    }, {
>
>      ...
>
>    }]
>
> }
>
> I’d be more interested in each piece of “failing or passing” content
> having it’s own meta – rather than using arrays of bunched data.  The
> reason being that adding an example should not break loads of other stuff,
> our cause a maintenance headache.
>
>
> Indeed, during the development of EARL we had looked at aggregation and
> encountered many problems. The statement above says that a web page has 4
> violations and 6 passes to three tests, but the mappings are unclear. The
> statement is difficult to process any further beyond this point.
>
> We’re also looking to automate the transpile of our unit test content into
> this format, and that is aided by having individual test cases defined
> separately.
>
> I would propose the following:
>
> {
>
>    "organisation": "",
>
>    "license": "",
>
>    "a11y-testcases": [
>
> {
>
>      "url" or "htmlFragment": "https://raw.githubusercontent.com/
> dequelabs/axe-core/master/test/integration/rules/image-
> alt/image-alt.html#monkeys",
>
>      "wcagRefs": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>
>      "conforms":false | true]
>
> },
>
> {
>
>    etc… per piece of content...
>
>
> This maps pretty nearly to EARL:
>
> - organisation -> earl:Assertor
> - url/htmlFragment -> earl:TestSubject
> - wcagRefs -> earl:TestCriterion
> - conforms -> TestResult
>
> The option to provide a url *or HTML fragment* would be very useful in
> terms of consuming other peoples data…
>
>
> An earl:TestSubject could be any of:
>
> - simple URI
> - "content" (simple text, Base64, or XML)
> - HTTP response (headers, and/or body)
> - "Document" (description of document)
> - "Software" (description of software)
>
> The “conforms” status is simply set to a true or false.
>
>
> An earl:TestResult consists of:
>
> - outcome (pass, fail, cannot tell, not tested, not applicable)
> - "info" (optional - simple text, for additional information)
> - "pointer" (optional - to identify location within TestSubject)
>
> I have also dropped the testId as it’s not necessary, and could make the
> test cases subject to change.
>
> Thoughts / comments welcome.  But, it would be good to get this
> nailed-down as soon as possible, so we can move ahead with publishing the
> test cases…
>
>
> I'm wondering if we should attempt an updated Working Group Note of EARL,
> only this time using JSON rather than RDF/XML serialization for the
> examples? It looks to me like everything is pretty much there.
>  - https://www.w3.org/TR/EARL10-Schema/
>
> Best,
>   Shadi
>
> Alistair
>
>
> --
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> Accessibility Strategy and Technology Specialist
> Web Accessibility Initiative (WAI)
> World Wide Web Consortium (W3C)
>
>
>
>
>
> --
>
> *Wilco Fiers*
>
> Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
>
>


-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG