Re: Notes about automating WAI-ARIA tests within the Web Platform Tests environment.

Thank you Jon.

Best,
Rich
> On Jun 24, 2016, at 12:18 PM, Gunderson, Jon R <jongund@illinois.edu> wrote:
> 
> These are some of the ideas I discussed with Shane McCarron today.  He introduced me to the “Move the Web Forward” project for testing web standards compliance in general.   
>  
> This framework of tests are used by browser developers to continuously monitor compliance, so if we can use this framework the ARIA testing could become a part of existing testing processes.
>  
> I will try to schedule a call for next week to discuss all the ideas we have collected.
>  
> I have also updated the wiki with the links he provided.
> https://www.w3.org/wiki/ARIA_1.1_Automated_Testing <https://www.w3.org/wiki/ARIA_1.1_Automated_Testing>
>  
> Jon
>  
>  
> From: Shane McCarron [mailto:shane@spec-ops.io <mailto:shane@spec-ops.io>] 
> Sent: Friday, June 24, 2016 12:07 PM
> To: Gunderson, Jon R <jongund@illinois.edu <mailto:jongund@illinois.edu>>
> Subject: Notes about automating WAI-ARIA tests within the Web Platform Tests environment.
>  
> Jon,
>  
> Great speaking with you.  Here are my thoughts (before I lose them):
>  
> WPT is the standard environment for W3C working groups to use for testing.  Especially groups that provide components of the "web platform" - which means web clients and web servers.  ARIA is clearly part of the web platform.  The WPT environment is well documented here [1].  It is the result of the 'test the web forward' effort at W3C [2].  You can see a public-facing test environment at [3]
>  
> Automated tests in WPT are run continuously by browser developers as part of their processes.
>  
> WPT has support for webdriver built in [4].  Also support for manual tests, reference tests, test automation via JS, server side manipulation of protocols via static files and via python, and a reporting framework that allows for easy generation and maintenance of implementation reports.
>  
> The basic architecture of a client side test is this:
>  
> WEB BROWSER   <----->  CHILD WINDOW
> MAIN WINDOW            FOR INDIVIDUAL TEST
> 
> The main window acts as the driver, opening up a child window and populating it with data from the web test server (HTML, JS, etc).  In fully automated tests, some JS then evaluates the DOM or the network activity or whatever, then decides if the test passed or failed.  The JS API is well documented here [5].
>  
> My concept for ARIA tests would augment this general flow like this:
>  
> WEB BROWSER   <----->  CHILD WINDOW
> MAIN WINDOW            FOR INDIVIDUAL TEST
>                              ^
>                              |
>                              v
>                         LOCAL AT SHIM  
>  
> The local AT shim is a "fake" assistive technology client that attaches the AT API of the platform under test - the one on which the browser is running.  It listens on the AT API for data and communicates it back to the child window via a websocket or some other well defined, standard interface that will work on the platform (note - if that is not possible, it could also communicate with the child window VIA the web platform test server - basically using it as a conduit to pass data back and forth).
>  
> The "tests" would do whatever the test needs to do (set focus on something, fill in an input field, etc). then get back what the AT layer generated and evaluate it - ALL in the child window for the individual test.  It would then report the pass or failure of the test along with any other relevant information. When all tests within a page complete, the test completes and the main window automatically cycles to the next test in the sequence.
>  
> The output of the main window is JSON that has basic information.  That information is used by reporting tools to maintain implementation reports that can help with CR criteria evaluation [6].  Test results reports and maintained by the w3c at [7]. A simple example report is at [8].  This is a good example as it is being constantly updated by the developers.
>  
> Note that you do not need to develop the JS tests by hand.  You could define a declarative grammar and then generate the ".html" files from that.  You can see an example of that approach in what we are doing for the Web Annotation Group currently at [9]
>  
> I hope this helps as an intro.  Feel free to circulate this to other players in the ARIA test space if you think it has merit.  I look forward to working with you!
>  
>  
> [1] https://wptserve.readthedocs.io/en/latest/index.html <https://urldefense.proofpoint.com/v2/url?u=https-3A__wptserve.readthedocs.io_en_latest_index.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=OfEOv5HUwuDpQc1mbhQSgBStMRcUKbClVihp-YWq7TM&e=>
> [2] http://testthewebforward.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__testthewebforward.org&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=UiMLkaac2u-Hl8UkqzGWDN5L0W6gubA-HEhdVYtvt1U&e=>
> [3] http://w3c-test.org/tools/runner/index.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__w3c-2Dtest.org_tools_runner_index.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=EkcJHUheACyviq8u9Jto05P3vJjgDvaUXCaTjYoGbNg&e=>
> [4] https://github.com/w3c/wptrunner <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_wptrunner&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=8ol8QqZNNBObLuB-uR61UjG7G4Nj58ak8Xia8t4yobg&e=>
> [5] http://testthewebforward.org/docs/testharness-library.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__testthewebforward.org_docs_testharness-2Dlibrary.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=sZPilczQh7wGIx-V7ztxbrd7XX7zz9j0CRoU4lFOS74&e=>
> [6] https://github.com/w3c/wptreport <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_wptreport&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=HletjxkIYzhum6NK0ZqZmfCmV1aJjuJtOn5gijuXshk&e=>
> [7] https://github.com/w3c/test-results/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_test-2Dresults_&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=_-wxAEYiYEr0B8EevgtFQuSQA-1vHIVhUM6bU-xFGLs&e=>
> [8] http://w3c.github.io/test-results/battery-status/all.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__w3c.github.io_test-2Dresults_battery-2Dstatus_all.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=ujX6L_E6ZFPJSrGv1aRd9Bz_9MTD1W2KHFa22RvFGZQ&e=> 
> [9] https://github.com/Spec-Ops/web-platform-tests/tree/master/annotation-model <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Spec-2DOps_web-2Dplatform-2Dtests_tree_master_annotation-2Dmodel&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=XqX-FysId78z1A8kCvUXWQSdRZNNQW9KF94bqd69ENc&e=>
> -- 
> Shane McCarron
> Projects Manager, Spec-Ops