RE: Meeting to discuss automated testing

John,

I have no meetings scheduled, but am putting together a wiki page on resources.
https://www.w3.org/wiki/ARIA_1.1_Automated_Testing


I will ask Rich to put on the agenda for the next ARIA meeting to discuss a meeting time, since it seems we have people from three difference continents who want to participant in the testing.

I will also report on the Thursday call my discussions with Microsoft on their open source automated testing technology.

I am currently trying to identify the automated technologies that will be available to use for testing, sine this is a new area for me.

Does Deque have any resources to help build automated ARIA implementation testing tools for OS X, this is our biggest need right now?

I am starting to work on a draft format for creating tests that we can then review as a group and work with the people actually building the automated testing tools to see if it will work for their technology.

Jon

P.S. Michael Cooper has reminded me that all communications on ARIA testing need to use the public-aria-testing@w3.org<mailto:public-aria-testing@w3.org> listserv.  Please use this in the future for any questions or comment.

From: John Foliot [mailto:john.foliot@deque.com]
Sent: Monday, June 20, 2016 10:43 AM
To: John Jansen <John.Jansen@microsoft.com>
Cc: Gunderson, Jon R <jongund@illinois.edu>; Cynthia Shelly <cyns@microsoft.com>; Rakesh Paladugula <rakesh.paladugula@deque.com>; Suman Damera <suman.damera@deque.com>; Rich Schwerdtfeger <richschwer@gmail.com>; Michael Cooper <cooper@w3.org>; Joanmarie Diggs <jdiggs@igalia.com>; Wilco Fiers <wilco.fiers@deque.com>
Subject: Re: Meeting to discuss automated testing

Hi all,

Has a specific time and date been set for this Friday call? I've spoken with one of our senior engineers, Wilco Fiers (who is also the Chair of the W3C Automated WCAG Monitoring Community Group) and he'd like to be able to sit in on this call as well.

Next steps? Do we have meeting details?

Thanks!

JF

On Mon, Jun 13, 2016 at 2:43 PM, John Jansen <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>> wrote:
Sure. I just sent a link to it via the other thread, but we can discuss it this week. My calendar is pretty full, but should be OK if before 10am PDT.

-----Original Message-----
From: Gunderson, Jon R [mailto:jongund@illinois.edu<mailto:jongund@illinois.edu>]
Sent: Monday, June 13, 2016 9:43 AM
To: John Jansen <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>>; Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>; 'Rakesh Paladugula' <rakesh.paladugula@deque.com<mailto:rakesh.paladugula@deque.com>>; 'Suman Damera' <suman.damera@deque.com<mailto:suman.damera@deque.com>>
Cc: 'Rich Schwerdtfeger' <richschwer@gmail.com<mailto:richschwer@gmail.com>>; 'Michael Cooper' <cooper@w3.org<mailto:cooper@w3.org>>; John Foliot <john.foliot@deque.com<mailto:john.foliot@deque.com>>; 'Joanmarie Diggs' <jdiggs@igalia.com<mailto:jdiggs@igalia.com>>
Subject: Re: Meeting to discuss automated testing

John and Cynthia,

Is there a time this week I could call you this week to learn more about your automated testing tool and the format it supports for consuming test cases?

I think it sounds like right now you have the most advanced tool and I would like to learn more about how it works.

Based on that discussion we could then have a more general discuss with the rest of the group.

Jon





On 5/20/16, 9:59 AM, "John Jansen" <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>> wrote:

>I think that sounds like a good idea. We already have a "Browser
>Testing and Tools" working group that is focused on WebDriver, but may
>be a good place to work as well. In other words, this may be the kind
>of thing that would be interesting to more groups than ARIA ...
>honestly, though, I'm not sure :-)
>
>*-----Original Message-----
>*From: Gunderson, Jon R [mailto:jongund@illinois.edu<mailto:jongund@illinois.edu>]
>*Sent: Friday, May 20, 2016 6:38 AM
>*To: John Jansen <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>>; John Foliot
>*<john.foliot@deque.com<mailto:john.foliot@deque.com>>; Cynthia Shelly <cyns@microsoft.com<mailto:cyns@microsoft.com>>; 'Rakesh
>*Paladugula' <rakesh.paladugula@deque.com<mailto:rakesh.paladugula@deque.com>>; 'Suman Damera'
>*<suman.damera@deque.com<mailto:suman.damera@deque.com>>
>*Cc: 'Rich Schwerdtfeger' <richschwer@gmail.com<mailto:richschwer@gmail.com>>; 'Joanmarie Diggs'
>*<jdiggs@igalia.com<mailto:jdiggs@igalia.com>>; 'Michael Cooper' <cooper@w3.org<mailto:cooper@w3.org>>
>*Subject: Re: Meeting to discuss automated testing
>*
>*John,
>*
>*I am proposing we start a sub group of the ARIA working group to begin
>*discussions on what tools are available and what we need to be able to
>test *implementation across multiple platforms.
>*
>*Look forward to your participation in the testing group.
>*
>*Jon
>*
>*
>*
>*
>*On 5/13/16, 5:17 PM, "John Jansen" <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>> wrote:
>*
>*>This is great Jon. Are you thinking this will be inperson before
>TPAC? We *might want to wait until Sept and book time to discuss then,
>or we could do it *sooner, but over IRC to avoid additional travel. I
>should likely have concrete *answers to how we are thinking about it at Microsoft in the coming weeks.
>*>
>*>> -----Original Message-----
>*>> From: Gunderson, Jon R [mailto:jongund@illinois.edu<mailto:jongund@illinois.edu>] *>> Sent:
>Friday, May 13, 2016 2:11 PM *>> To: John Foliot
><john.foliot@deque.com<mailto:john.foliot@deque.com>>; Cynthia Shelly *>> <cyns@microsoft.com<mailto:cyns@microsoft.com>>; John
>Jansen <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>>; *>> 'Rakesh Paladugula'
><rakesh.paladugula@deque.com<mailto:rakesh.paladugula@deque.com>>; 'Suman Damera'
>*>> <suman.damera@deque.com<mailto:suman.damera@deque.com>>
>*>> Cc: 'Rich Schwerdtfeger' <richschwer@gmail.com<mailto:richschwer@gmail.com>>; 'Joanmarie Diggs'
>*>> <jdiggs@igalia.com<mailto:jdiggs@igalia.com>>; 'Michael Cooper' <cooper@w3.org<mailto:cooper@w3.org>> *>> Subject:
>Meeting to discuss automated testing *>> *>> I have asked Michael
>Cooper to provide me with information on how to *>> organize a W3C
>hosted meeting to discuss testing.
>*>>
>*>> Here is a draft agenda:
>*>> 1. The current technologies available to automatically run test
>cases *>> on browsers and export results
>*>>    a. Ideally we would want open source or public tools to be able to
>*>> consume test cases, execute the tests and export the results 2.
>*>> Identify a data format that automated tools:
>*>>    a. Identify the location of test cases
>*>>    b. Identify expected results
>*>>    c. Export results
>*>> 3. Current capabilities of the W3C Test Harness for supporting *>>
>automated testing 4. Current set of test cases in the harness *>> *>>
>Jon *>> *>> *>> *>> -----Original Message----- *>> From: John Foliot
>[mailto:john.foliot@deque.com<mailto:john.foliot@deque.com>] *>> Sent: Thursday, April 28, 2016 1:15
>PM *>> To: 'Cynthia Shelly' <cyns@microsoft.com<mailto:cyns@microsoft.com>>; 'John Jansen'
>*>> <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>>; 'Rakesh Paladugula'
>*>> <rakesh.paladugula@deque.com<mailto:rakesh.paladugula@deque.com>>; 'Suman Damera'
>*>> <suman.damera@deque.com<mailto:suman.damera@deque.com>>
>*>> Cc: 'Rich Schwerdtfeger' <richschwer@gmail.com<mailto:richschwer@gmail.com>>; 'Joanmarie Diggs'
>*>> <jdiggs@igalia.com<mailto:jdiggs@igalia.com>>; 'Michael Cooper' <cooper@w3.org<mailto:cooper@w3.org>>; Gunderson,
>Jon *>> R <jongund@illinois.edu<mailto:jongund@illinois.edu>> *>> Subject: RE: Automated
>accessibility testing *>> *>> Hi Cynthia, *>> *>> I am including both
>Suman Damera and Rakesh Paladugula from  our *>> Indian office on this
>response - they have joined the ARIA WG with *>> the desire to (amongst
>other things) write test scripts and assist *>> with ARIA 1.1 testing,
>and so they will likely need to be added to *>> your white-list. I am
>also copying Jon Gunderson who was (I believe) *>> looking at getting
>ARIA 1.1 tests written and out, and who chatted *>> with Rakesh at CSUN
>about moving forward/next steps (although I do not *have any specific
>details there).
>*>>
>*>> Suman, Rakesh, please contact John Jansen at Microsoft and provide
>*>> the info required (github ID). I am sure that he or Cynthia can
>also *>> answer any specific questions on how to move forward.
>*>> Rich/Michael/Cynthia, if there is anything else I need to do
>W3C-wise to *get Rakesh and Suman set up, please advise.
>*>> Otherwise, I will step back. (and yes Cynthia, this is great news!)
>*>> *>> Cheers!
>*>>
>*>> JF
>*>>
>*>>
>*>> > -----Original Message-----
>*>> > From: Cynthia Shelly [mailto:cyns@microsoft.com<mailto:cyns@microsoft.com>] *>> > Sent:
>Thursday, April 28, 2016 1:05 PM *>> > To: John Jansen
><John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>>; Joanmarie Diggs *>> > <jdiggs@igalia.com<mailto:jdiggs@igalia.com>>;
>John Foliot <john.foliot@deque.com<mailto:john.foliot@deque.com>>; Michael *>> > Cooper
><cooper@w3.org<mailto:cooper@w3.org>> *>> > Cc: Rich Schwerdtfeger <richschwer@gmail.com<mailto:richschwer@gmail.com>> *>>
>> Subject: Automated accessibility testing *>> > *>> > TL;DR *>> > If
>you want to start looking at and using our accessibility test *>>
>automation, reply *>> > to John Jansen <John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>> with
>your github id, and *>> > then *>> go *>> > to
>https://github.com/MicrosoftEdge/A11Y<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftEdge_A11Y&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=oTKfyZlx7k7v1BSJ0pS4PyoxHNsEIKn1AhqgdnZi3BE&s=dAwuNKOYelGzrZTrVVjH34ewzikEDhL2dmh3C_SLvss&e=>
>*>> >
>*>> > Please keep this to yourselves for now, until we've finished the
>*>> > process *>> of *>> > making this code more widely available.
>*>> >
>*>> > John F,
>*>> > We can also add you or someone on your staff to this. It will be
>*>> > more *>> widely *>> > available soon, but there are still some
>corporate hoops to jump *>> > through *>> to *>> > publish it. I
>thought it would be better to get going on ARIA test *>> creation
>sooner *>> > than later.
>*>> >
>*>> > To run this, you'll need the most recent Windows 10 Insider Build
>*>> > and *>> Visual *>> > Studio (free community version is fine). The
>code as it is will run *>> > the *>> test files on *>> >
>html5accessibiltiy.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__html5accessibiltiy.com&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=oTKfyZlx7k7v1BSJ0pS4PyoxHNsEIKn1AhqgdnZi3BE&s=3eHNehsmGqEX_0Oqf4-ag4WTkhIiX3BXNNKptwfkA10&e=> with Edge. That should be enough to see how *>>
>> to *>> create *>> > test files.
>*>> >
>*>> > Michael,
>*>> > I'm hoping to get as much accessibility test collateral as
>possible *>> available to this *>> > system. Can we leverage the
>existing ARIA 1.0 tests? How hard would *>> > it be *>> to do *>> >
>auto-generated test cases for ARIA 1.1? Core-AAM and HTML-AAM?
>*>> > AccName- AAM seems harder to auto-generate, given its algorithmic
>*>> > nature. Are there *>> test *>> > cases for that?
>*>> >
>*>> > This system pulls tests from github. John J thought this would be
>a *>> > good *>> > location:
>*>> > https://github.com/w3c/web-platform-tests/<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_web-2Dplatform-2Dtests_&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=oTKfyZlx7k7v1BSJ0pS4PyoxHNsEIKn1AhqgdnZi3BE&s=sIunwW9FfL1ZaxvosszzwkS9oyJ_dEfhnVsk2OYoMYE&e=>
>*>> > [as an example: https://github.com/w3c/web-platform-<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_web-2Dplatform-2D&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=oTKfyZlx7k7v1BSJ0pS4PyoxHNsEIKn1AhqgdnZi3BE&s=HeY9nbyLvh5fd9ui5m2Elkw6ShqVfV-YbJnYW799Biw&e=>
>*>> > tests/tree/master/cssom-view]
>*>> >
>*>> > The format is somewhat different. Do you think it would be easier
>*>> > to port *>> the *>> > existing test cases, or change the
>auto-generation templates to *>> > match the *>> new *>> >
>requirements?
>*>> >
>*>> > I'm so excited to see this actually start to happen!
>*>> >
>*>> > Thanks,
>*>> > Cynthia
>*>> >
>*>> > -----Original Message-----
>*>> > From: John Jansen
>*>> > Sent: Wednesday, April 27, 2016 12:10 PM *>> > To: Joanmarie
>Diggs <jdiggs@igalia.com<mailto:jdiggs@igalia.com>>; Cynthia Shelly *>> > <cyns@microsoft.com<mailto:cyns@microsoft.com>>
>*>> > Subject: RE: Introduction *>> > *>> > Hey Joanie, *>> > *>> > I
>am so sorry for the delay. While Microsoft is really opening up, *>> >
>we *>> still *>> > sometimes have to talk to lawyers about stuff :-)
>*>> > *>> > If you send me your github user name, I can add you to the
>private *>> > repo we *>> have *>> > that allows us to automate
>html5accessibility.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__html5accessibility.com&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=oTKfyZlx7k7v1BSJ0pS4PyoxHNsEIKn1AhqgdnZi3BE&s=zSseNMVSLiodIV5l1R-80dLJvNR6mWJHRJlRS4kPeUo&e=>. We also have a *>> > sample project up there
>that will work with generic webpages. I *>> > think there is *>> enough
>*>> > info in the readme and code that you will find interesting. I'd
>*>> > love to *>> get your *>> > feedback, as well as have you
>contribute back to it.
>*>> >
>*>> > https://github.com/MicrosoftEdge/A11Y<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftEdge_A11Y&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=oTKfyZlx7k7v1BSJ0pS4PyoxHNsEIKn1AhqgdnZi3BE&s=dAwuNKOYelGzrZTrVVjH34ewzikEDhL2dmh3C_SLvss&e=>
>*>> >
>*>> > -John
>*>> >
>*>> >
>*>> > *-----Original Message-----
>*>> > *From: Joanmarie Diggs [mailto:jdiggs@igalia.com<mailto:jdiggs@igalia.com>] *>> > *Sent:
>Tuesday, March 1, 2016 2:26 PM *>> > *To: Cynthia Shelly
><cyns@microsoft.com<mailto:cyns@microsoft.com>>; John Jansen *>> > *<John.Jansen@microsoft.com<mailto:John.Jansen@microsoft.com>>
>*>> > *Subject: Re: Introduction *>> > * *>> > *Thanks Cynthia!
>*>> > *
>*>> > *John: If you have anything you could share (even a *>> >
>work-in-progress, *>> don't *>> > *show anyone thing) that would be
>great. I'll look around for my *>> > proof-of- *concept hacks from last
>year.
>*>> > *
>*>> > *--joanie
>*>> > *
>*>> > *On 03/01/2016 04:21 PM, Cynthia Shelly wrote:
>*>> > *> Both of you are working on automating accessibility testing,
>and *>> > it *> *>> seems *>> > like you have pretty similar
>approaches. You should talk J *>> > * *>> *>



--
John Foliot
Principal Accessibility Strategist
Deque Systems Inc.
john.foliot@deque.com<mailto:john.foliot@deque.com>

Advancing the mission of digital accessibility and inclusion