Update on ARIA Automation Progress

(apologies, Jon asked me to put this into the wiki space, but I couldn't
decide where and frankly this was faster.)

tl;dr: It works!

=======

For the last several weeks, Joanie and I have been working on a solution
for automating testing of ARIA.  The live design space for the solution was
in the wiki at [1].

A couple of weeks ago I started the process of implementing the solution as
proposed, and it went very well.  Joanie then jumped in and implemented a
component that permits the testing of ATK-based user agents on Linux.  That
also went very well.  So much so that we were able to demonstrate working
ARIA 1.1 testing of Firefox to the ARIA working group in Lisbon. You can
see the presentation at [5].

While doing this implementation work, I decided that we should have a
formal specification for how these pieces fit together that could be used
as guidance for future working groups / activities. That evolving
specification is at [2].

In order to make certain that the path from "testable statement" to actual
automated test case is as straight and frictionless as possible, it was
necessary to change the format of the wiki(s) that we use for these
statements, such as the one at [3].  The format is documented in the
overview section.  I have written a tool to take existing wikis in the
"old" format and convert them to this new, normalized format.  I continue
to refine that tool.

The working implementation of the framework is part of the WPT
environment.  It is currently in a branch on the Spec-Ops fork of WPT repo
on github [4] (aria-atta-support) in the folder wai-aria.  Configuration of
this environment is not super difficult, but it is not the work of moments.


Joanie has done amazing work with the Linux platform support.  Her work is
in a separate branch of the same repo [5] and her development work is
ongoing.  As of yesterday, she indicated that in addition being able to
test Firefox, she was also testing through a WebKit based browser on the
same platform, using the same ATTA.  This is an excellent proof of concept
for us.  Go Joanie!

What's left to do?

   1. Add "linting" to the test generator so that it will complain if there
   are errors in testable statements.
   2. Clean up the "manual mode" support in the tests.
   3. Experiment with using "wptrunner" to drive the tests.
   4. Finish converting the ARIA 1.1 Wiki testable statements to the new
   format.  My script should be able to do the first pass, but the tests need
   careful review by platform experts.
   5. Fill in any gaps in the tests.
   6. Decide if we want to test state changes as part of this effort
   (discuss at meeting)
   7. Get ATTA implementations for IAccessible2 / MSAA (Jon), UIA
   (Stanley), and NSAccessiblility (Joanie?)

Outside of the ARIA 1.1 effort, I think we can easily do the following:

   1. Convert the SVG2 Testable Statements
   2. Convert the ARIA 1.0 Testable Statements
   3. Integrate HTML5 A11Y tests from Steve Faulkner
   4. Integrate the ATTAs with WPT so they can be used by browser
   developers all the time
   5. See about ATTA for Android (possible) / iOS (less likely)
   6. Write a tool that knows the AAM mappings and can generate the
   Testable Statements from generic ARIA assertions (super-stretch goal)

That's all for this morning. I look forward to speaking with each of you
this afternoon.


[1] https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Proposed_Architecture
[2] https://spec-ops.github.io/atta-api/index.html
[3] https://www.w3.org/wiki/ARIA_1.1_Testable_Statements#Overview
[4] https://github.com/Spec-Ops/web-platform-tests/tree/aria-atta-support
[5]
https://docs.google.com/presentation/d/1-28KVLhJ6M0RDA96RQG7yn8Mxvfau3mssmQuuogIx98/edit?usp=sharing
-- 
Shane McCarron
Projects Manager, Spec-Ops