MINUTES: 31 August 2016 ARIA Automated Testing Sub Group Teleconference

Link:
https://www.w3.org/2016/08/31-aria-test-minutes.html



[W3C]<http://www.w3.org/>

- DRAFT -
Accessible Rich Internet Applications Working Group Teleconference
31 Aug 2016

See also: IRC log<http://www.w3.org/2016/08/31-aria-test-irc>

Attendees

Present
Joanmarie_Diggs, fesch, Rich_Schwerdtfeger, jongund
Regrets
Chair
jongund
Scribe
cynthia, cyns
Contents

  *   Topics<https://www.w3.org/2016/08/31-aria-test-minutes.html#agenda>
     *   test case updates<https://www.w3.org/2016/08/31-aria-test-minutes.html#item01>
     *   architecture<https://www.w3.org/2016/08/31-aria-test-minutes.html#item02>
  *   Summary of Action Items<https://www.w3.org/2016/08/31-aria-test-minutes.html#ActionSummary>
  *   Summary of Resolutions<https://www.w3.org/2016/08/31-aria-test-minutes.html#ResolutionSummary>

________________________________

<fesch> https://www.w3.org/dpub/docs/scribing-guidelines.xhtml


<jongund> trackbot, start meeting

<ShaneM> is the code in the topic correct?

<jongund> Scribe: cynthia

<jongund> Meeting: ARIA Testing

<cyns> scribe: cyns

<jongund> scribe: cyns

test case updates

<jongund> https://www.w3.org/wiki/ARIA_1.1_Testable_Statements


jg: do we have any update on that?

fe: gave state and property statements to John F's team
... working on script that turns wiki tables to json object

<jongund> https://www.w3.org/wiki/ARIA_1.1_Testable_Statements_JSON


fe: this may be hard to use when there is a complex tree
... looking for feedback on how usable the json is

https://www.w3.org/wiki/ARIA_1.1_Testable_Statements_JSON


jg: is there validation?

fe: no

jg: script available?

fe: will post soon

jg: is this ok for shane?

sm: yes

jg: will find out what John F and team is doing

<jongund> https://www.w3.org/wiki/index.php?title=ARIA_1.1_Testable_Statements&action=history


architecture

jg: talked to shane about wpt solution strategy
... also microsoft strategy
... key to both is accessibility api shims that reflect back accessibility info to testing framework
... is the shim self contained in msft architecture?
... I have a student who is going to start working on msaa/ia2 shim
... what shane needed is pretty simple

db: it is self contained. it runs locally. spins up pages through web driver, then evaluates against requirements.
... requirements are currently hard-coded, but will be bringing in from json.
... looking at output in a way wpt can consume
... we have api's in c# that allow us to talk to uia, which then talks to the browser.
... when you talk to uia, what is it giving you back? a11y tree info?
... yes, accessibility tree objects

jg: what change is needed to use ia2 in your driver?

db: designed with other api/browser in mind. it's factored out.

jg: shane, what is your approach

sm: wpt architecture is complicated. because it's desinged to run in places where it's hard to do anything cool, like locally running code

<jongund> https://www.w3.org/wiki/ARIA_1.1_Automated_Testing


sm: wpt has test web pages with test code embedded within them.

<jongund> https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Proposed_Architecture


sm: can use webdriver but does't need it
... can be a lot of interaction between test server and child window
... idea is to have child window talk to wpt server as a relay to talk to the shim
... can use in differnt environemtns, like android

cs: not needed for aria 1.1

sm: true, but good for futre
... is msft tool pulling expected results from child window?

db: not yet

sm: do we want evaluative smarts in test window, so they have to be written once, or in logic that is part of platform specific
... is it doing the evaluation?

db: yes

sm: are you oepn to pulling the logic up into a javascript library that might live in the test?

db: I think that's possible, but a lot of work
... client code can run anywhere c# can be compiled (which is most platforms)

sm: in the diagram, see "magic". Is it possible for get the needed info from the client window?

db: yes, but file would be easier

sm: easier for me, maybe not for you... if the local at shim had a port, then child test window could instruct it what to do from localhost

db: that makes sense to me

sm: that would let me remove the server-side part on the right

cs: details of "magic"

sm: RESTful api, with window title and criteria, and yes or no.
... wpt would call that subtests

<ShaneM> individual results for each assertion evaluated per test window.

sm: in wpt world, when result came back, test runner can tell system to open next test

jg: would that make the microsoft and wpt tools more compatible

db: yes. changing to take input from a port rather than file shouldn't be too hard

sm: like that
... my goal is that this effort is useful going forward for testing user agents as they are developed, like the rest of wpt
... the easier it is for browser developers to know when aria is broken and fix it, the better
... stanley, can we look at integration points

sh: yes

jd: sounds interesting, but need to look at code

sm: there is some setup for full-scale automated testing. you'll have to start up the shim and the browser.

jd: that's easy

(discussion of configuration requirements for setting up wpt)

jg: I have a student who will be working on IA2. she's looking at wpt now and accprobe.
... next step will be looking at microsoft test harness

db: already prototyped

cs: branch is public

<dbrett> https://github.com/microsoftEdge/a11y/tree/aria


sm: I'll do the work for the wpt pieces

db: it can read any json

fe: I can tweak too

Summary of Action Items
Summary of Resolutions
[End of minutes]
________________________________
Minutes formatted by David Booth's scribe.perl<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/scribedoc.htm> version 1.144 (CVS log<http://dev.w3.org/cvsweb/2002/scribe/>)
$Date: 2016/08/31 21:47:44 $
________________________________
Scribe.perl diagnostic output
[Delete this section before finalizing the minutes.]

This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/>

Guessing input format: RRSAgent_Text_Format (score 1.00)

Found Scribe: cynthia
Found Scribe: cyns
Found Scribe: cyns
Inferring ScribeNick: cyns
Scribes: cynthia, cyns
Present: Joanmarie_Diggs fesch Rich_Schwerdtfeger jongund
Found Date: 31 Aug 2016
Guessing minutes URL: http://www.w3.org/2016/08/31-aria-test-minutes.html

People with action items:

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.




[End of scribe.perl<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/scribedoc.htm> diagnostic output]

Jon Gunderson, Ph.D.
Coordinator
Accessible IT Group
Disability Resources and Education Services
College of Applied Health Sciences
University of Illinois at Urbana-Champaign

E-mail: jongund@illinois.edu
WWW: http://disability.illinois.edu/academic-support/aitg