Re: Test case format (explicit results)

Hi Alistair,
I think a metadata format for test cases isn't part of the work statement
for this group. It would have been a nice addition, but it's looking to be
a lot more complicated then I initially expected it was going to be. We
just had a talk about it in the ACT planning call. I've put the topic on
the agenda for this week. We may just want to move this work into another
group. There has already been talk about revisiting EARL. This might be
better suited in that discussion.

Wilco

On Mon, Dec 18, 2017 at 4:49 PM, Alistair Garrison <
alistair.garrison@levelaccess.com> wrote:

> Hi,
>
>
>
> Isn’t the format for presenting test content against a known outcome part
> of “9.1 Implementation Validation”?
>
>
>
> If yes, don’t we have to agree a format in this group?  Shadi’s suggested
> EARL format, for example, would I think also fit nicely with other pieces
> of the work – like test results, and the EARL aggregation model used in
> Appendix 1…
>
>
>
> Maybe I’m wrong?
>
>
>
> Alistair
>
>
>
> *From: *Wilco Fiers <wilco.fiers@deque.com>
> *Date: *Monday, 18 December 2017 at 15:04
>
> *To: *Alistair Garrison <alistair.garrison@levelaccess.com>
> *Cc: *Shadi Abou-Zahra <shadi@w3.org>, WCAG ACT TF <public-wcag-act@w3.org
> >
> *Subject: *Re: Test case format (explicit results)
>
>
>
> Hi Alistair,
>
> Apologies. I didn't mean to imply this JSON structure was ever formally
> voted on. It's not a deliverable for ACT TF, it's just something that would
> be a nice bonus it it could be done in a way that supports the development
> of the ACT Rules format. We used this to base the test cases for the ACT
> Rules examples on. Those were voted on before we published them. For
> instance this one:
>
> https://w3c.github.io/wcag-act-rules/rule-tests/ACT-R2-tests.html
>
>
>
> If you want to track the discussion, I looked up some of the minutes for
> you:
>
> https://www.w3.org/2017/09/25-wcag-act-minutes.html
>
> https://www.w3.org/2017/06/05-wcag-act-minutes.html
>
> https://www.w3.org/2017/05/15-wcag-act-minutes.html
>
> https://www.w3.org/2017/04/24-wcag-act-minutes.html
>
>
>
>
>
> W
>
>
>
> On Mon, Dec 18, 2017 at 3:06 PM, Alistair Garrison <alistair.garrison@
> levelaccess.com> wrote:
>
> Hi Wilco,
>
>
>
> I’m sorry – I can’t remember the ACT-group vote on that format, maybe I
> was away – I’ll happily check it out.  Can you send me a link to the vote,
> or minutes containing the vote?
>
>
>
> All the best
>
>
>
> Alistair
>
>
>
> *From: *Wilco Fiers <wilco.fiers@deque.com>
> *Date: *Monday, 18 December 2017 at 12:54
> *To: *Alistair Garrison <alistair.garrison@levelaccess.com>
> *Cc: *Shadi Abou-Zahra <shadi@w3.org>, WCAG ACT TF <public-wcag-act@w3.org
> >
>
>
> *Subject: *Re: Test case format (explicit results)
>
>
>
> Hi Alistair,
>
> The format that's there now has been agreed on by participants from Deque,
> IBM and SiteImprove. This work was planned and finished months ago, and
> we've already put time into the implementation of it. I think if things
> need to be added to it, that's fine, but to ask us to redo our
> implementation work because there are features that you *don't* need,
> that's not something I'm keen on unless there are tangible problems with it.
>
>
>
> As for the links, we created a list with several test case repositories.
> They are on the same page as the proposed format: https://www.w3.org/
> WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources#
> Accessibility_Testing_Repositories
>
>
>
> W
>
>
>
> On Mon, Dec 18, 2017 at 11:36 AM, Alistair Garrison <alistair.garrison@
> levelaccess.com> wrote:
>
> Hi Wilco,
>
>
>
> We don’t use ids in our own test content – as we use HTML Fragments,
> within Jasmine unit tests.  The reasons we avoid ids being that:
>
>
>
> ·         I don’t like the use of ids to identify test content, as ids
> can of course form part of the material / mechanisms being tested [I was
> illustrating with your aXe unit test, which do seemingly use ids…  so not
> using ids is fine with me…]
>
>
>
> ·         I also don’t like the clustering of multiple pieces of test
> content into a single test page – as bleed is always possible between
> pieces of content;
>
>
>
> ·         I also don’t like “passed” or “failed” as fields, as I’ve
> already noted – as this is merely there I suspect to support using blocks
> of test content identified with ids written into a single page…
>
>
>
> Can you point to an example of Va11ys content, or Bentoweb content so we
> can see how they are presented?
>
>
>
> We don’t of course need to agree the format that our unit test content is
> published in – as it is the act of publishing them publically that is the
> main thing.  I suspect, however, that we’d simply opt for using Shadi’s
> suggestion, as it seems to closely map to what we do already… It also is in
> line with the previous EARL work, which I believe this group might “do
> something with” going forward…
>
>
>
> Alistair
>
>
>
> *From: *Wilco Fiers <wilco.fiers@deque.com>
> *Date: *Monday, 18 December 2017 at 10:08
> *To: *Shadi Abou-Zahra <shadi@w3.org>
> *Cc: *Alistair Garrison <alistair.garrison@levelaccess.com>, WCAG ACT TF <
> public-wcag-act@w3.org>
> *Subject: *Re: Test case format (explicit results)
>
>
>
> Hi Alistair, Shadi,
>
>
>
> If you output your data like this, you can have them as single items.
> There is nothing that prevents you from having the URL multiple times. It
> just create a bunch of duplication.
>
>
>
> {
>
>   "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>   "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>
>   "passed": ["#pass1"]
>
> }, {
>
>   "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>   "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>
>   "passed": ["#pass2"]
>
> }, ...
>
>
>
> As for your proposal. I don't think this works for Va11ys, or for the
> Bentoweb project. Neither of those uses IDs to point to their results. The
> Bentoweb examples use XPath (which is easily transformed to CSS), and IBM
> doesn't have selectors for Va11ys. One of the goals for this was to allow
> it to work for existing libraries. What you're proposing doesn't seem to do
> that, where as the existing proposal does allow your use case.
>
>
>
>  W
>
>
>
> On Mon, Dec 18, 2017 at 7:58 AM, Shadi Abou-Zahra <shadi@w3.org> wrote:
>
> Hi,
>
>
> On 15/12/2017 15:19, Alistair Garrison wrote:
>
> I’m interested in altering the Test case format (explicit results) model.
> We have the following on the table at present:
>
> {
>
>    "name": "axe-core",
>
>    "version": "2.1.7",
>
>    "license": "MPL-2.0",
>
>    "a11y-testcases": [{
>
>      "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/
> test/integration/rules/image-alt/image-alt.html",
>
>      "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html",
> "mytool:image-alt"],
>
>      "failed": ["#violation1", "#violation2", "#violation3",
>
> "#violation4"],
>
>      "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5",
>
> "#pass6"]
>
>    }, {
>
>      ...
>
>    }]
>
> }
>
> I’d be more interested in each piece of “failing or passing” content
> having it’s own meta – rather than using arrays of bunched data.  The
> reason being that adding an example should not break loads of other stuff,
> our cause a maintenance headache.
>
>
> Indeed, during the development of EARL we had looked at aggregation and
> encountered many problems. The statement above says that a web page has 4
> violations and 6 passes to three tests, but the mappings are unclear. The
> statement is difficult to process any further beyond this point.
>
> We’re also looking to automate the transpile of our unit test content into
> this format, and that is aided by having individual test cases defined
> separately.
>
> I would propose the following:
>
> {
>
>    "organisation": "",
>
>    "license": "",
>
>    "a11y-testcases": [
>
> {
>
>      "url" or "htmlFragment": "https://raw.githubusercontent.com/
> dequelabs/axe-core/master/test/integration/rules/image-
> alt/image-alt.html#monkeys",
>
>      "wcagRefs": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html"],
>
>      "conforms":false | true]
>
> },
>
> {
>
>    etc… per piece of content...
>
>
> This maps pretty nearly to EARL:
>
> - organisation -> earl:Assertor
> - url/htmlFragment -> earl:TestSubject
> - wcagRefs -> earl:TestCriterion
> - conforms -> TestResult
>
> The option to provide a url *or HTML fragment* would be very useful in
> terms of consuming other peoples data…
>
>
> An earl:TestSubject could be any of:
>
> - simple URI
> - "content" (simple text, Base64, or XML)
> - HTTP response (headers, and/or body)
> - "Document" (description of document)
> - "Software" (description of software)
>
> The “conforms” status is simply set to a true or false.
>
>
> An earl:TestResult consists of:
>
> - outcome (pass, fail, cannot tell, not tested, not applicable)
> - "info" (optional - simple text, for additional information)
> - "pointer" (optional - to identify location within TestSubject)
>
> I have also dropped the testId as it’s not necessary, and could make the
> test cases subject to change.
>
> Thoughts / comments welcome.  But, it would be good to get this
> nailed-down as soon as possible, so we can move ahead with publishing the
> test cases…
>
>
> I'm wondering if we should attempt an updated Working Group Note of EARL,
> only this time using JSON rather than RDF/XML serialization for the
> examples? It looks to me like everything is pretty much there.
>  - https://www.w3.org/TR/EARL10-Schema/
>
> Best,
>   Shadi
>
> Alistair
>
>
> --
> Shadi Abou-Zahra - http://www.w3.org/People/shadi/
> Accessibility Strategy and Technology Specialist
> Web Accessibility Initiative (WAI)
> World Wide Web Consortium (W3C)
>
>
>
>
>
> --
>
> *Wilco Fiers*
>
> Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
>
>
>
>
>
> --
>
> *Wilco Fiers*
>
> Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
>
>
>
>
>
> --
>
> *Wilco Fiers*
>
> Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG
>
>


-- 
*Wilco Fiers*
Senior Accessibility Engineer - Co-facilitator WCAG-ACT - Chair Auto-WCAG