MINUTES: 24 August 2016 ARIA Automated Testing Sub Group Teleconference

Link:
https://www.w3.org/2016/08/24-aria-test-minutes.html


[W3C]<http://www.w3.org/>

- DRAFT -
Accessible Rich Internet Applications Working Group Teleconference
24 Aug 2016

Agenda<https://lists.w3.org/Archives/Public/public-aria-test/2016Aug/0008.html>

See also: IRC log<http://www.w3.org/2016/08/24-aria-test-irc>

Attendees
Present
Joanmarie_Diggs, jongund, JF, cyns, ShaneM
Regrets
Chair
jongund
Scribe
ShaneM
Contents

  *   Topics<https://www.w3.org/2016/08/24-aria-test-minutes.html#agenda>
     *   Test Case Format<https://www.w3.org/2016/08/24-aria-test-minutes.html#item01>
     *   Automation Platform<https://www.w3.org/2016/08/24-aria-test-minutes.html#item02>
  *   Summary of Action Items<https://www.w3.org/2016/08/24-aria-test-minutes.html#ActionSummary>
  *   Summary of Resolutions<https://www.w3.org/2016/08/24-aria-test-minutes.html#ResolutionSummary>

________________________________

<jongund> trackbot, start meeting

<jongund> Meeting: ARIA Testing

https://www.w3.org/wiki/ARIA_1.1_Testable_Statements#alert

<scribe> Scribe: ShaneM

<scribe> Chair: joanie

<scribe> Chair: jongund

Test Case Format

fred: putting things in the testable statement document
... using tables

There are some gaps. I don't know what a pattern is or what it means to say something is not in a pattern.

<jongund> https://www.w3.org/wiki/ARIA_1.1_Testable_Statements

cyns: I didn't understand the question

fesch: Under some statements it says "not in a certain pattern"

cyns: Yes, that does mean something...

<fesch> https://www.w3.org/wiki/ARIA_1.1_Testable_Statements#menuitemradio

ShaneM: It may well, but not to any sort of automation system. We don't know what patterns are.

fesch: Go look at "cell"

<jongund> https://www.w3.org/wiki/ARIA_1.1_Testable_Statements#cell

fesch: it says "selection pattern TODO" or "invoke pattern TODO"

cyns: basically it means that the invoke pattern is implemented with the default values

ShaneM: if there were a pointer to what an invoke pattern is, then we are laughing.

cyns: I will provide a link to the documentation

<cyns> https://msdn.microsoft.com/en-us/library/windows/desktop/ee671194%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

fesch: describes the layout

will get rolled up into a JSON object.

as long as it is interpretable by whatever the automation framework is, we should be good

jongund: Why do we need MSAA?

fesch: because it is in the table.

jongund: do we really need to support this? It seems to be very limited.

cyns: an MSAA only implementation would be largely unusable.

fesch: if there is a platform that is mixed, then we would need to evaluate both in order to cover

cyns: Edge is only UIA only

fesch: IAcessible2 is what Firefox. MSAA is legacy. IAccessible2 is not a MS product that is implemented in FF and Chrome and maybe others. UIA is the current MS A11Y API.

jongund: question about "if given"

fesch: that's not used at all in the conversion
... as far as I know this is how the testable statements are supposed to be written

ShaneM: I recommend we maintain it

jongund: the information is in the table

ShaneM: then... no need to duplicate

jongund: if it is something we are not using then it is not adding any new information.

fesch: it doesn't add any value

cyns: the information in the "then" clause should be translated into the table too

fesch: I have put notes in about information that is missing.
... rich gave me some information about where to find other information in a branch. that's mroe than I wanted to do right now.

jongund: should the test include information about these non-inherited attributes?

fesch: The reason they are in there because this is new.

ShaneM: I dont think it matters

cyns: you would want to know which thing failed

ShaneM: In WPT each assertion comes out in its own subtest result so you get the information regardless

cyns: You can use the role without the properties. They are not required.

jongund: Are browsers required to compute indexes or is it just a should?
... most browsers don't do it automatically. So ATs need to do it independently.
... What do we need to do to get out of ARIA 1.1 CR testing. The tests should be minimal.
... would it make more sense for MS to have multiple tests?

???: what would this mean if there were multiple tests?

jongund: each code snippet will generate a test file.
... Does MS framework need separate tests to get results?

joanie: I am waiting for a group decision as to direction before I put in too much work.

jongund: we are going to talk about that next.

fesch: there are thigns not in the Core AAM and it would be great if we could get them in.
... colcount, rowcount, some roles.

joanie: all the new stuff is on rich's branch and it not yet merged because some people have not signed off on it yet.

fesch: it is just hard to do things without the branch merged.
... there are some where there is no change in the mappings. So there is no ARIA 1.1 testing needed.
... but I might be wrong.
... Some things support a couple of new attributes. When I made a test I made the role with the new attributes. For example the only thing to test for menuitem is with the attributes...

<JF> +1 to cyns

fesch: we are only checking the deltas...

JF: we may apply a little reason and logic to the question of "what is tested when something changed from 1.0"

cyns: it seems a little easier to just enumerate the possibilities for the changes roles than trying to figure out what is changed.

JF: I sort of agree with Shane that we should ensure things aren't broken. Especially since it is automated.

fesch: Does anyone have trouble with the format as it is?

cyns: the only question is "do we want to maintain wiki tables"? Would a nested list be easier to maintain?

JF: I find it easier to look at HTML than wiki tables

fesch: The problem is that some tables are uneven.

cyns: I am going to look for a wysiwyg table editor

fesch: its a lot of bookkeeping

jongund: we need to move on...
... fesch you got started on this. Can some of JF people can get into this now?

JF: Rich has some statements that need to be done. We are going to look at that, then feed it back to the people in India, and then see how they do with a few example

fesch: all of section 3 is availabhle to be done. aria properties and state testable statements. fesch is working on 4.
... if someone wants to work on "none". I think it would be weird to have only one.

cyns: yes. It needs multiple.

jongund: Can JF pleease share with the list what people are wokring on so there are no collisions or wasted effort.
... tomorrow on the ARIA call maybe we can sync up.

Automation Platform

jongund: MS is committed to their tech but only for UIA
... where are we at for other platforms? joanie is committed to implementing on ATK.

<joanie> https://www.w3.org/WAI/ARIA/wiki/ARIA_1.1_Changes

joanie: the testable statements page is based upon the change log.

This other page is way more detailed. That's the one that has all of the good information.

scribe: I basically did a giant diff to build that other page.

jongund: does that page need to be merged?

joanie: you need to at least look at it very carefully.

fesch: who created the testable statements page?

jongund: I did.
... I will go through and integrate joanie work.
... thank you for all that work joanie

ShaneM: I thought we were going to try to get WPT to drive tests

cyns: I thought we were going to use the MS stuff and attempt to integrate with WPT reporting if we had time.

jongund: ShaneM do you have examples? We dont have a communication channel between the AT layer and the test window.
... Are there things in the changes to aria 1.1 that would require more complex "interaction" or state change testing?

joanie: There might be things. I don't know.

cyns: We didnt test them in 1.0 so why would we in 1.1?

jongund: there are some events that we might need to test. Does UIA have those?
... we are not going to finalize this plan today. can we meet next week?

ShaneM: let's continue the conversatin via email.

jongund: let's talk in a week.

Summary of Action Items
Summary of Resolutions
[End of minutes]
________________________________
Minutes formatted by David Booth's scribe.perl<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/scribedoc.htm> version 1.144 (CVS log<http://dev.w3.org/cvsweb/2002/scribe/>)
$Date: 2016/08/24 22:01:57 $
________________________________
Scribe.perl diagnostic output
[Delete this section before finalizing the minutes.]

This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34

Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/>



Guessing input format: RRSAgent_Text_Format (score 1.00)



Succeeded: s/jongund/joanie/

Succeeded: s/interate/integrate/

Found Scribe: ShaneM

Inferring ScribeNick: ShaneM

Present: Joanmarie_Diggs jongund JF cyns ShaneM

Agenda: https://lists.w3.org/Archives/Public/public-aria-test/2016Aug/0008.html

Found Date: 24 Aug 2016

Guessing minutes URL: http://www.w3.org/2016/08/24-aria-test-minutes.html

People with action items:


[End of scribe.perl<http://dev.w3.org/cvsweb/%7Echeckout%7E/2002/scribe/scribedoc.htm> diagnostic output]