MINUTES: 13 July 2016 ARIA Automated Implementation Testing Sub Group Teleconference

Link:
https://www.w3.org/2016/07/13-aria-test-minutes.html



[W3C]<http://www.w3.org/>

- DRAFT -
Accessible Rich Internet Applications Working Group Teleconference
13 Jul 2016

See also: IRC log<http://www.w3.org/2016/07/13-aria-test-irc>

Attendees

Present
ShaneM, Cyns, jongund_, Joanmarie_Diggs, Rich
Regrets
Chair
Jon Gunderson
Scribe
jongund_
Contents

  *   Topics<https://www.w3.org/2016/07/13-aria-test-minutes.html#agenda>
     *   What we are doing with test automation for ARIA<https://www.w3.org/2016/07/13-aria-test-minutes.html#item01>
  *   Summary of Action Items<https://www.w3.org/2016/07/13-aria-test-minutes.html#ActionSummary>
  *   Summary of Resolutions<https://www.w3.org/2016/07/13-aria-test-minutes.html#ResolutionSummary>

________________________________

Hi Joanie

is the code

What we are doing with test automation for ARIA

<ShaneM> Some notes at wiki: https://www.w3.org/wiki/ARIA_1.1_Automated_Testing


<dbrett> https://github.com/MicrosoftEdge/A11y


Shane: The WPT can hold the tests
... How it interfaces with AT, my assumption was acting as an AT

David: yes

Shane: The concept is to make a channel communication between AT and the test framework
... The WPT has a framework for reporting to W3C

CS: The W3C has a report framework
... A way to communicate the AT and the test page with some Javascript
... The information can then be passed/fail
... We output CSV

Shane: There are security issues with local file system, browser have work arounds
... Providing the fake AT a URL handler with the results
... Pass the data in any useful format
... Goal is to inspect the result automatically
... We want the tests to be run automatically during normal browser development testing

CS: We care about more than CR
... We don't need W3C CR reports everytime

Shane: The webkit team uses this, the data is more than just reports
... Edge maybe using the WPT already, they are in the IRC channel

CS: Somebody

Shane: Joanie has been thinking of this for other APIs

JD: Narrow it down for me

Shane: Linux

JD: Yes, accessibility APIs
... Each platform would provide their own API

CS: Can you try to do this

Shane: David how do you feel about the code, can it send code

David: Yes I think it could be extended

Shane: It is strictly browser testing, no capability in turning on the AT connector
... You would need to start the AT emulator, then the browser
... Does AT attach to a window or how does it work

David: The AT looks at the Accessibility API to find edge

Shane: If there is a named window the AT could find the right window

David: You can just start with the top node and then start looking for the right window

JD: One event we get is a document load event, you can key off that event

Shane: In WPT test is a load, so there would be a load event

JD: That is like ideal, the first thing you need is the document and the event tells you it ready

RS: One question is there an MSAA?

David: There was initially, but we didn't create all the tests for MSAA
... There are old commits

JG: The WPT is feasible?

Shane: Let's talk about ARIA 1.0, how many tests?

RS: ~900

Shane: They were generally automatically generated?

CS: The expected results though where human readible
... I was planning on working on a format

RS: Some tests required a time delay, did you allow for that

David: Yes, we can handle waiting

CS: We will start with the simpler tests

Shane: WPT and its test runner framework, has an interesting async test module, based on events, so it can handle delays
... The battery test API is an example, so you have to wait for the battery percentage changes
... Let's not reinvent

CS: The format on the wiki looks OK, but is too simple

RS: You doing a comparison with a predicted states and properties, can you test events

CS: Is there anything else other than live regions
... Can we test events?

David: It is possible

Shane: How do events get exposed?

CS: There are accessibility API event and then start plateform events

<Rich> http://rawgit.com/w3c/aria/master/core-aam/core-aam.html#mapping_events


RS: State change events...
... You have text remove..., live regions...., there are special ones for UIA
... MSAA/IA2 they are pretty standard

CS: It is a feature, but there are many other things more important

RS: Agree

CS: We can work on that in the future

RS: They are pretty simple tests

JD: Event testing should be easy, if we are listening for events for knowing when to check, then we are already testing the events

Shane: There are 2 kinds of test, declarative states and properties, there are more complex tests that evaluates ....
... It sounds like the current test only sends back one file

CS: Event testing will be a different kind of test

<ShaneM> David: There is a way to get the data back iteratively.

JD: I will be automating the even test, I think this will be easy
... If this is going to change the harness, then that would be a priority

David: DIfferent branch for each browser and platform

JD: We need to figure out exactly needs to be done

Shane: SpecOps has some resources for this, what I can volunteer to do is get the infrastructure in place
... If we set this up so there is one click to start, if the test requires a change .....
... The low hanging fruit is where we should start

CS: We can do some interaction testing, but that is more work

Shane: If you are using web drive it works across browsers

David: It is just navigate and open

CS: We tested the date controls, but are more work to write

Shane: WPT is more than reporting, it will also a runner framework

CS: I will be looking at test cases, and port them to the fields

Shane: RS put a pointer in to core AAM
... Aren't the assertions in the document

JD: We have a chicken and egg issue

RS: ARIA 1.1 process to CR ....

JG: Next call in two weeks

Shane: We should be looking at next time having this machine extractable

JD: Is what the core AAM does is ... the checkbox is checked is is different on each platform

<ShaneM> ACTION: ShaneM to create sample wrapper to talk with a fake AT under WPT by 7/27 [recorded in http://www.w3.org/2016/07/13-aria-test-minutes.html#action01]

<trackbot> Created ACTION-2095 - Create sample wrapper to talk with a fake at under wpt by 7/27 [on Shane McCarron - due 2016-07-20].

<cyns> ATK connector (Joanie) IA2 connector AX Connector Event testing (Joanie) Reporting with WPT (Shane) Document test case format (Cyns) Parsing JSON test case format (Brett) Create tests (working group)

Summary of Action Items
[NEW] ACTION: ShaneM to create sample wrapper to talk with a fake AT under WPT by 7/27 [recorded in http://www.w3.org/2016/07/13-aria-test-minutes.html#action01]

Summary of Resolutions
[End of minutes]
________________________________
Minutes formatted by David Booth's scribe.perl<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/scribedoc.htm> version 1.144 (CVS log<http://dev.w3.org/cvsweb/2002/scribe/>)
$Date: 2016/07/13 22:05:50 $
________________________________
Scribe.perl diagnostic output
[Delete this section before finalizing the minutes.]

This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/>

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/aria//
No ScribeNick specified.  Guessing ScribeNick: jongund_
Inferring Scribes: jongund_
Present: ShaneM Cyns jongund_ Joanmarie_Diggs Rich
Found Date: 13 Jul 2016
Guessing minutes URL: http://www.w3.org/2016/07/13-aria-test-minutes.html

People with action items: shanem

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.




[End of scribe.perl<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/scribedoc.htm> diagnostic output]


Jon Gunderson, Ph.D.
Coordinator
Accessible IT Group
Disability Resources and Education Services
College of Applied Health Sciences
University of Illinois at Urbana-Champaign

E-mail: jongund@illinois.edu
WWW: http://disability.illinois.edu/academic-support/aitg