- From: Shane McCarron <shane@spec-ops.io>
- Date: Fri, 24 Jun 2016 17:45:15 -0500
- To: "Gunderson, Jon R" <jongund@illinois.edu>
- Cc: "public-aria-test@w3.org" <public-aria-test@w3.org>
- Message-ID: <CAJdbnOCf5to7D6Rab3qGwOs2q6q+5Y2U8wacN3WvS59O+74BFg@mail.gmail.com>
Just to be clear, this summary was something I wrote up in a few minutes after speaking with Jon this morning. There is a wealth of information on WPT at the end of those links. I have not done a thorough analysis of simplistic architecture I sketched out. And worst of all, I am on holiday as of the 28th. So I will try to monitor this whilst I am away, but... I may not respond as quickly as I would normally. On Fri, Jun 24, 2016 at 12:18 PM, Gunderson, Jon R <jongund@illinois.edu> wrote: > These are some of the ideas I discussed with Shane McCarron today. He > introduced me to the “Move the Web Forward” project for testing web > standards compliance in general. > > > > This framework of tests are used by browser developers to continuously > monitor compliance, so if we can use this framework the ARIA testing could > become a part of existing testing processes. > > > > I will try to schedule a call for next week to discuss all the ideas we > have collected. > > > > I have also updated the wiki with the links he provided. > > https://www.w3.org/wiki/ARIA_1.1_Automated_Testing > > > > Jon > > > > > > *From:* Shane McCarron [mailto:shane@spec-ops.io] > *Sent:* Friday, June 24, 2016 12:07 PM > *To:* Gunderson, Jon R <jongund@illinois.edu> > *Subject:* Notes about automating WAI-ARIA tests within the Web Platform > Tests environment. > > > > Jon, > > > > Great speaking with you. Here are my thoughts (before I lose them): > > > > WPT is the standard environment for W3C working groups to use for > testing. Especially groups that provide components of the "web platform" - > which means web clients and web servers. ARIA is clearly part of the web > platform. The WPT environment is well documented here [1]. It is the > result of the 'test the web forward' effort at W3C [2]. You can see a > public-facing test environment at [3] > > > > Automated tests in WPT are run continuously by browser developers as part > of their processes. > > > > WPT has support for webdriver built in [4]. Also support for manual > tests, reference tests, test automation via JS, server side manipulation of > protocols via static files and via python, and a reporting framework that > allows for easy generation and maintenance of implementation reports. > > > > The basic architecture of a client side test is this: > > > > WEB BROWSER <-----> CHILD WINDOW > MAIN WINDOW FOR INDIVIDUAL TEST > > > The main window acts as the driver, opening up a child window and > populating it with data from the web test server (HTML, JS, etc). In fully > automated tests, some JS then evaluates the DOM or the network activity or > whatever, then decides if the test passed or failed. The JS API is well > documented here [5]. > > > > My concept for ARIA tests would augment this general flow like this: > > > > WEB BROWSER <-----> CHILD WINDOW > MAIN WINDOW FOR INDIVIDUAL TEST > > ^ > > | > > v > > LOCAL AT SHIM > > > > The local AT shim is a "fake" assistive technology client that attaches > the AT API of the platform under test - the one on which the browser is > running. It listens on the AT API for data and communicates it back to the > child window via a websocket or some other well defined, standard interface > that will work on the platform (note - if that is not possible, it could > also communicate with the child window VIA the web platform test server - > basically using it as a conduit to pass data back and forth). > > > > The "tests" would do whatever the test needs to do (set focus on > something, fill in an input field, etc). then get back what the AT layer > generated and evaluate it - ALL in the child window for the individual > test. It would then report the pass or failure of the test along with any > other relevant information. When all tests within a page complete, the test > completes and the main window automatically cycles to the next test in the > sequence. > > > > The output of the main window is JSON that has basic information. That > information is used by reporting tools to maintain implementation reports > that can help with CR criteria evaluation [6]. Test results reports and > maintained by the w3c at [7]. A simple example report is at [8]. This is a > good example as it is being constantly updated by the developers. > > > > Note that you do not need to develop the JS tests by hand. You could > define a declarative grammar and then generate the ".html" files from > that. You can see an example of that approach in what we are doing for the > Web Annotation Group currently at [9] > > > > I hope this helps as an intro. Feel free to circulate this to other > players in the ARIA test space if you think it has merit. I look forward > to working with you! > > > > > > [1] https://wptserve.readthedocs.io/en/latest/index.html > <https://urldefense.proofpoint.com/v2/url?u=https-3A__wptserve.readthedocs.io_en_latest_index.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=OfEOv5HUwuDpQc1mbhQSgBStMRcUKbClVihp-YWq7TM&e=> > > [2] http://testthewebforward.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__testthewebforward.org&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=UiMLkaac2u-Hl8UkqzGWDN5L0W6gubA-HEhdVYtvt1U&e=> > > [3] http://w3c-test.org/tools/runner/index.html > <https://urldefense.proofpoint.com/v2/url?u=http-3A__w3c-2Dtest.org_tools_runner_index.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=EkcJHUheACyviq8u9Jto05P3vJjgDvaUXCaTjYoGbNg&e=> > > [4] https://github.com/w3c/wptrunner > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_wptrunner&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=8ol8QqZNNBObLuB-uR61UjG7G4Nj58ak8Xia8t4yobg&e=> > > [5] http://testthewebforward.org/docs/testharness-library.html > <https://urldefense.proofpoint.com/v2/url?u=http-3A__testthewebforward.org_docs_testharness-2Dlibrary.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=sZPilczQh7wGIx-V7ztxbrd7XX7zz9j0CRoU4lFOS74&e=> > > [6] https://github.com/w3c/wptreport > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_wptreport&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=HletjxkIYzhum6NK0ZqZmfCmV1aJjuJtOn5gijuXshk&e=> > > [7] https://github.com/w3c/test-results/ > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_w3c_test-2Dresults_&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=_-wxAEYiYEr0B8EevgtFQuSQA-1vHIVhUM6bU-xFGLs&e=> > > [8] http://w3c.github.io/test-results/battery-status/all.html > <https://urldefense.proofpoint.com/v2/url?u=http-3A__w3c.github.io_test-2Dresults_battery-2Dstatus_all.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=ujX6L_E6ZFPJSrGv1aRd9Bz_9MTD1W2KHFa22RvFGZQ&e=> > > > [9] > https://github.com/Spec-Ops/web-platform-tests/tree/master/annotation-model > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Spec-2DOps_web-2Dplatform-2Dtests_tree_master_annotation-2Dmodel&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=j6RBXrhEaYMLtCYE9QTLa61PNWdyrISJ8vgATwGPJFI&s=XqX-FysId78z1A8kCvUXWQSdRZNNQW9KF94bqd69ENc&e=> > > -- > > Shane McCarron > > Projects Manager, Spec-Ops > -- Shane McCarron Projects Manager, Spec-Ops
Received on Friday, 24 June 2016 22:46:12 UTC