- From: Dylan Barrell <dylan.barrell@deque.com>
- Date: Fri, 15 Jul 2016 14:49:14 -0400
- To: "Gunderson, Jon R" <jongund@illinois.edu>
- Cc: Shane McCarron <shane@spec-ops.io>, David Brett <dbrett@microsoft.com>, "public-aria-test@w3.org" <public-aria-test@w3.org>, public-test-infra <public-test-infra@w3.org>
- Message-ID: <CAKGs1QZySo2dm8-K3zB-MDuQockuNuaCrHotF=oqAWuy4U025w@mail.gmail.com>
That spec is orthogonal to the questions I raised. --Dylan On Fri, Jul 15, 2016 at 2:47 PM, Gunderson, Jon R <jongund@illinois.edu> wrote: > Dylan, > > The test pages will be written in HTML, CSS and Javascript, a JSON object > int he page will be used to identify platform specific mapping informaiton. > > Here is a link to a DRAFT of proposed format for a typical test page: > > <https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Draft_ARIA_Test_Case_Format> > https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Draft_ARIA_Test_Case_Format > > > Jon > > > From: Dylan Barrell <dylan.barrell@deque.com> > Date: Friday, July 15, 2016 at 1:34 PM > To: Shane McCarron <shane@spec-ops.io> > Cc: David Brett <dbrett@microsoft.com>, "public-aria-test@w3.org ARIA > Implementation Testing" <public-aria-test@w3.org>, Jon Gunderson < > jongund@illinois.edu>, public-test-infra <public-test-infra@w3.org> > Subject: Re: Communicating between the WPT test window and a separate > application on the client > > I would like some clarifying information: > > 1) Do you have an opinion on what language the tests will be written in? > Would it be a requirement to have this be bindable to any language or do we > just propose picking one? > 2) In your diagrams, there is some control instance running the browsers. > In the Web testing world, the communication would normally be along the > lines of > > TEST RUNNER <-----------> BROWSER AUTOMATION (e.g. WebDriver) > <-----------> BROWSER > > How does this picture map to the picture(s) you drew above? Could it be > seen as follows > <-----------> BROWSER AUTOMATION (e.g. > WebDriver) <-----------> BROWSER > TEST RUNNER > <-----------> LOCAL AT SHIM <-----------> OS > > Where the test runner is responsible for communicating with both the > browser automation AND the AT shim? > > If this is the case, then the logic for executing individual or batch > tests is inside the test runner and could be seen as essentially an > expansion of the WebDriver API. If this is the case, it could be bound to > any language including JS (Node), Java, etc. > > --Dylan > > > On Fri, Jul 15, 2016 at 1:50 PM, Shane McCarron <shane@spec-ops.io> wrote: > >> I'm out of position right now. I understand what you are proposing, but I >> think it flies in the face of the general architecture of tests under WPT. >> Basically in the w3c you need to have individual tests for each feature and >> those should be individually executable. I can imagine how to accomplish >> that in your model... Let me ponder a little more. >> >> On Jul 15, 2016 12:40 PM, "David Brett" <dbrett@microsoft.com> wrote: >> >> I’d like to suggest another architecture that could be simpler. Instead >> of creating a line of communication between the AT and the main browser >> window for each test, we could instead pass the entire list of files (and >> requirements) at once, let the same code that manages the AT handle running >> through the tests, and then pass all the data back at the end. Here is my >> own (terrible ASCII art) diagram: >> >> >> >> WEB BROWSER >> >> MAIN WINDOW >> >> ^ >> >> | >> >> MAGIC >> >> | >> >> v CHILD WINDOW >> >> LOCAL AT SHIM <-----> FOR INDIVIDUAL TESTS >> >> >> >> Obviously we have to make that “magic” connection only twice instead of >> hundreds of times which will most likely be more reliable and save time. I >> don’t think this will result in any extra work if we use A11y, since the >> system is already set up to handle a list of tests. >> >> >> >> As far as how that communication works, I think it would be pretty >> straightforward to include the test requirement files in a directory of the >> WPT repo and let A11y iterate over those. Sending the results through a >> websocket once the tests are completed shouldn’t be too hard either. >> >> >> >> *From:* Gunderson, Jon R [mailto:jongund@illinois.edu] >> *Sent:* Friday, July 15, 2016 7:30 AM >> *To:* Shane McCarron <shane@spec-ops.io>; public-test-infra < >> public-test-infra@w3.org>; public-aria-test@w3.org >> *Subject:* RE: Communicating between the WPT test window and a separate >> application on the client >> >> >> >> Shane thank you again for getting this conversation started and your >> interest in getting ARIA testing as part of WPT. >> >> >> >> Just for clarification. The “fake AT” is basically a application that >> exposes information about a platform specific accessibility API as WPT or >> some other tool exercises a ARIA test page. So the application needs to >> look the accessibility tree and related events and get that information >> back into the browser for comparison with expected results for us to >> utilize the WPT framework to its fullest extent. >> >> >> >> Jon >> >> >> >> >> >> *From:* Shane McCarron [mailto:shane@spec-ops.io <shane@spec-ops.io>] >> *Sent:* Friday, July 15, 2016 8:43 AM >> *To:* public-test-infra <public-test-infra@w3.org>; >> public-aria-test@w3.org >> *Subject:* Communicating between the WPT test window and a separate >> application on the client >> >> >> >> Hi! >> >> >> >> The ARIA Working Group is investigating various ways to automate the >> testing of ARIA - which requires testing the accessibility api and its >> communication with assistive technologies (AT) on a bunch of platforms. >> Obviously, this is a bit of a challenge. The current thinking is that a >> fake AT can be provided on each platform. The fake AT is started by the >> tester (or test automation environment) prior to starting a test run. Once >> it is running and has found the test window, it will capture the >> accessibility tree and events as the tests set up and manipulate the DOM. >> Simple enough. >> >> >> >> Except, of course, for getting the information from the fake AT back into >> the test window. Consider the following (terrible ASCII art) diagram: >> >> >> >> WEB BROWSER <-----> CHILD WINDOW >> >> MAIN WINDOW FOR INDIVIDUAL TEST >> >> ^ >> >> | >> >> MAGIC >> >> | >> >> v >> >> LOCAL AT SHIM >> >> >> >> The "MAGIC" is where I am playing right now. Here are some of my ideas: >> >> 1. Shim is passed a URI on the WPT server on startup (or finds the >> URI when it finds the test window). Communicates with it through a >> websocket, and the window in which the test is running communicates with >> the same websocket endpoint. Data is relayed that way. This seems the >> most portable to me. >> 2. Shim runs a simple HTTP listener. Child window communicates with >> that using HTTP (websocket or simple HTTP GET) on localhost. This requires >> implementing a messaging stack... which doesn't feel very easy on every >> platform but is probably do-able. It might also violate CORS stuff, but >> again - do-able. >> 3. Rely on some sort of native messaging that is platform specific. >> This doesn't feel scalable to me. It would also mean modifying the WPT >> part of this any time we wanted to add another platform that had a >> different native messaging capability. >> 4. Use a ServiceWorker in some magical way that I probably don't >> understand. Feels like a steep learning curve. Also, they don't seem to >> be widely supported yet. >> >> My hope is that some of you have already thought about a similar problem >> (there is a native application running on the platform under test that >> needs to send messages into the test in order to evaluate success or >> failure). So... any ideas? >> >> -- >> >> Shane McCarron >> >> Projects Manager, Spec-Ops >> >> >> > > > -- > Download the aXe browser extension for free: > > Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools > <https://urldefense.proofpoint.com/v2/url?u=https-3A__addons.mozilla.org_en-2DUS_firefox_addon_axe-2Ddevtools&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=_JVA5omJG9Ir99XWLuJRR0feRTWZKLYVYxlNXWGn_pA&s=Labt5u56v6mvY5zOGP3drcLE1gWLoTNMGcfTGH4QbvE&e=> > Chrome: > https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US > <https://urldefense.proofpoint.com/v2/url?u=https-3A__chrome.google.com_webstore_detail_axe_lhdoppojpmngadmnindnejefpokejbdd-3Fhl-3Den-2DUS&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=_JVA5omJG9Ir99XWLuJRR0feRTWZKLYVYxlNXWGn_pA&s=VeYI_KeRTXk6sXG4Vt24RShA1lVJ7T2tZMHqCwPdJho&e=> > > Life is ten percent what happens to you and ninety percent how you respond > to it. - Lou Holtz > > -- Download the aXe browser extension for free: Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools Chrome: https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US Life is ten percent what happens to you and ninety percent how you respond to it. - Lou Holtz
Received on Friday, 15 July 2016 18:49:45 UTC