Re: Communicating between the WPT test window and a separate application on the client from Gunderson, Jon R on 2016-07-15 (public-test-infra@w3.org from July to September 2016)

From: Gunderson, Jon R <jongund@illinois.edu>
Date: Fri, 15 Jul 2016 19:27:53 +0000
To: Dylan Barrell <dylan.barrell@deque.com>
CC: Shane McCarron <shane@spec-ops.io>, David Brett <dbrett@microsoft.com>, "public-aria-test@w3.org" <public-aria-test@w3.org>, public-test-infra <public-test-infra@w3.org>
Message-ID: <F257B98B-7B9C-49F2-AA0C-A8A87D0A612E@illinois.edu>
Dylan,

At least with the WPT method Javascript would be the preferred way to communicate with the AT emulator.

I believe Microsoft is written in C# and uses WebDriver to control the browser.

Jon


From: Jon Gunderson <jongund@illinois.edu<mailto:jongund@illinois.edu>>
Date: Friday, July 15, 2016 at 2:01 PM
To: Dylan Barrell <dylan.barrell@deque.com<mailto:dylan.barrell@deque.com>>
Cc: Shane McCarron <shane@spec-ops.io<mailto:shane@spec-ops.io>>, David Brett <dbrett@microsoft.com<mailto:dbrett@microsoft.com>>, "public-aria-test@w3.org<mailto:public-aria-test@w3.org> ARIA Implementation Testing" <public-aria-test@w3.org<mailto:public-aria-test@w3.org>>, public-test-infra <public-test-infra@w3.org<mailto:public-test-infra@w3.org>>
Subject: Re: Communicating between the WPT test window and a separate application on the client
Resent-From: "public-aria-test@w3.org<mailto:public-aria-test@w3.org> ARIA Implementation Testing" <public-aria-test@w3.org<mailto:public-aria-test@w3.org>>
Resent-Date: Friday, July 15, 2016 at 2:02 PM

Dylan,

The way using WPT framework (which is what is used to test the implementation of web standards compliance in user agents) would be the test page would need some way to get the results of a specific test, so the page needs to communicate with the AT Emulator is someway that is accessible to the browser.

Microsoft has implemented a testing framework where the test result information is processed outside the browser, so the framework controls both the browser and the AT emulator.

You can see the wiki to get details on both approaches.  Microsoft has a proven approach and WPT there are questions on whether an AT emulator can communicate with a test page in some way.

The WPT way is preferable (if possible) because then the ARIA implementation test could be run with all the other WPT tests and use the same reporting system.

Jon


From: Dylan Barrell <dylan.barrell@deque.com<mailto:dylan.barrell@deque.com>>
Date: Friday, July 15, 2016 at 1:49 PM
To: Jon Gunderson <jongund@illinois.edu<mailto:jongund@illinois.edu>>
Cc: Shane McCarron <shane@spec-ops.io<mailto:shane@spec-ops.io>>, David Brett <dbrett@microsoft.com<mailto:dbrett@microsoft.com>>, "public-aria-test@w3.org<mailto:public-aria-test@w3.org> ARIA Implementation Testing" <public-aria-test@w3.org<mailto:public-aria-test@w3.org>>, public-test-infra <public-test-infra@w3.org<mailto:public-test-infra@w3.org>>
Subject: Re: Communicating between the WPT test window and a separate application on the client

That spec is orthogonal to the questions I raised.

--Dylan


On Fri, Jul 15, 2016 at 2:47 PM, Gunderson, Jon R <jongund@illinois.edu<mailto:jongund@illinois.edu>> wrote:
Dylan,

The test pages will be written in HTML, CSS and Javascript, a JSON object int he page will be used to identify platform specific mapping informaiton.

Here is a link to  a DRAFT of proposed format for a typical test page:
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_wiki_ARIA-5F1.1-5FAutomated-5FTesting-23Draft-5FARIA-5FTest-5FCase-5FFormat&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=1dXTpkaxcJAWFHXRulvC3AXNO7ZVwl0gYUvi1fz6Kqs&s=lpQdsW0k-tAYVEMbIHEywFRiuc7K90gvB01TnRJSeug&e=>https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Draft_ARIA_Test_Case_Format<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.w3.org_wiki_ARIA-5F1.1-5FAutomated-5FTesting-23Draft-5FARIA-5FTest-5FCase-5FFormat&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=1dXTpkaxcJAWFHXRulvC3AXNO7ZVwl0gYUvi1fz6Kqs&s=lpQdsW0k-tAYVEMbIHEywFRiuc7K90gvB01TnRJSeug&e=>

Jon


From: Dylan Barrell <dylan.barrell@deque.com<mailto:dylan.barrell@deque.com>>
Date: Friday, July 15, 2016 at 1:34 PM
To: Shane McCarron <shane@spec-ops.io<mailto:shane@spec-ops.io>>
Cc: David Brett <dbrett@microsoft.com<mailto:dbrett@microsoft.com>>, "public-aria-test@w3.org<mailto:public-aria-test@w3.org> ARIA Implementation Testing" <public-aria-test@w3.org<mailto:public-aria-test@w3.org>>, Jon Gunderson <jongund@illinois.edu<mailto:jongund@illinois.edu>>, public-test-infra <public-test-infra@w3.org<mailto:public-test-infra@w3.org>>
Subject: Re: Communicating between the WPT test window and a separate application on the client

I would like some clarifying information:

1) Do you have an opinion on what language the tests will be written in? Would it be a requirement to have this be bindable to any language or do we just propose picking one?
2) In your diagrams, there is some control instance running the browsers. In the Web testing world, the communication would normally be along the lines of

TEST RUNNER <-----------> BROWSER AUTOMATION (e.g. WebDriver) <-----------> BROWSER

How does this picture map to the picture(s) you drew above? Could it be seen as follows
                           <-----------> BROWSER AUTOMATION (e.g. WebDriver) <-----------> BROWSER
TEST RUNNER
                           <-----------> LOCAL AT SHIM <-----------> OS

Where the test runner is responsible for communicating with both the browser automation AND the AT shim?

If this is the case, then the logic for executing individual or batch tests is inside the test runner and could be seen as essentially an expansion of the WebDriver API. If this is the case, it could be bound to any language including JS (Node), Java, etc.

--Dylan


On Fri, Jul 15, 2016 at 1:50 PM, Shane McCarron <shane@spec-ops.io<mailto:shane@spec-ops.io>> wrote:

I'm out of position right now. I understand what you are proposing, but I think it flies in the face of the general architecture of tests under WPT.  Basically in the w3c you need to have individual tests for each feature and those should be individually executable.  I can imagine how to accomplish that in your model... Let me ponder a little more.

On Jul 15, 2016 12:40 PM, "David Brett" <dbrett@microsoft.com<mailto:dbrett@microsoft.com>> wrote:
I’d like to suggest another architecture that could be simpler. Instead of creating a line of communication between the AT and the main browser window for each test, we could instead pass the entire list of files (and requirements) at once, let the same code that manages the AT handle running through the tests, and then pass all the data back at the end. Here is my own (terrible ASCII art) diagram:

WEB BROWSER
 MAIN WINDOW
     ^
     |
   MAGIC
     |
     v                     CHILD WINDOW
LOCAL AT SHIM  <----->  FOR INDIVIDUAL TESTS

Obviously we have to make that “magic” connection only twice instead of hundreds of times which will most likely be more reliable and save time. I don’t think this will result in any extra work if we use A11y, since the system is already set up to handle a list of tests.

As far as how that communication works, I think it would be pretty straightforward to include the test requirement files in a directory of the WPT repo and let A11y iterate over those. Sending the results through a websocket once the tests are completed shouldn’t be too hard either.

From: Gunderson, Jon R [mailto:jongund@illinois.edu<mailto:jongund@illinois.edu>]
Sent: Friday, July 15, 2016 7:30 AM
To: Shane McCarron <shane@spec-ops.io<mailto:shane@spec-ops.io>>; public-test-infra <public-test-infra@w3.org<mailto:public-test-infra@w3.org>>; public-aria-test@w3.org<mailto:public-aria-test@w3.org>
Subject: RE: Communicating between the WPT test window and a separate application on the client

Shane thank you again for getting this conversation started and your interest in getting ARIA testing as part of WPT.

Just for clarification.  The “fake AT” is basically a application that exposes information about a platform specific accessibility API as WPT or some other tool exercises a ARIA test page.   So the application needs to look the accessibility tree and related events and get that information back into the browser for comparison with expected results for us to utilize the WPT framework to its fullest extent.

Jon


From: Shane McCarron [mailto:shane@spec-ops.io]
Sent: Friday, July 15, 2016 8:43 AM
To: public-test-infra <public-test-infra@w3.org<mailto:public-test-infra@w3.org>>; public-aria-test@w3.org<mailto:public-aria-test@w3.org>
Subject: Communicating between the WPT test window and a separate application on the client

Hi!

The ARIA Working Group is investigating various ways to automate the testing of ARIA - which requires testing the accessibility api and its communication with assistive technologies (AT) on a bunch of platforms.  Obviously, this is a bit of a challenge.  The current thinking is that a fake AT can be provided on each platform.  The fake AT is started by the tester (or test automation environment) prior to starting a test run.  Once it is running and has found the test window, it will capture the accessibility tree and events as the tests set up and manipulate the DOM.  Simple enough.

Except, of course, for getting the information from the fake AT back into the test window.  Consider the following (terrible ASCII art) diagram:


 WEB BROWSER   <----->  CHILD WINDOW

 MAIN WINDOW            FOR INDIVIDUAL TEST

                              ^

                              |

                            MAGIC

                              |

                              v

                         LOCAL AT SHIM

The "MAGIC" is where I am playing right now.  Here are some of my ideas:

  1.  Shim is passed a URI on the WPT server on startup (or finds the URI when it finds the test window). Communicates with it through a websocket, and the window in which the test is running communicates with the same websocket endpoint.  Data is relayed that way.  This seems the most portable to me.
  2.  Shim runs a simple HTTP listener.  Child window communicates with that using HTTP (websocket or simple HTTP GET) on localhost.  This requires implementing a messaging stack... which doesn't feel very easy on every platform but is probably do-able.  It might also violate CORS stuff, but again - do-able.
  3.  Rely on some sort of native messaging that is platform specific.  This doesn't feel scalable to me.  It would also mean modifying the WPT part of this any time we wanted to add another platform that had a different native messaging capability.
  4.  Use a ServiceWorker in some magical way that I probably don't understand.  Feels like a steep learning curve.  Also, they don't seem to be widely supported yet.
My hope is that some of you have already thought about a similar problem (there is a native application running on the platform under test that needs to send messages into the test in order to evaluate success or failure).  So... any ideas?
--
Shane McCarron
Projects Manager, Spec-Ops




--
Download the aXe browser extension for free:

Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools<https://urldefense.proofpoint.com/v2/url?u=https-3A__addons.mozilla.org_en-2DUS_firefox_addon_axe-2Ddevtools&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=_JVA5omJG9Ir99XWLuJRR0feRTWZKLYVYxlNXWGn_pA&s=Labt5u56v6mvY5zOGP3drcLE1gWLoTNMGcfTGH4QbvE&e=>
Chrome: https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US<https://urldefense.proofpoint.com/v2/url?u=https-3A__chrome.google.com_webstore_detail_axe_lhdoppojpmngadmnindnejefpokejbdd-3Fhl-3Den-2DUS&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=_JVA5omJG9Ir99XWLuJRR0feRTWZKLYVYxlNXWGn_pA&s=VeYI_KeRTXk6sXG4Vt24RShA1lVJ7T2tZMHqCwPdJho&e=>

Life is ten percent what happens to you and ninety percent how you respond to it. - Lou Holtz




--
Download the aXe browser extension for free:

Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools<https://urldefense.proofpoint.com/v2/url?u=https-3A__addons.mozilla.org_en-2DUS_firefox_addon_axe-2Ddevtools&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=1dXTpkaxcJAWFHXRulvC3AXNO7ZVwl0gYUvi1fz6Kqs&s=euYb-PKBjmPqEXtPXiSaFNh6xVYWzFNp5hY8f3wamW0&e=>
Chrome: https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US<https://urldefense.proofpoint.com/v2/url?u=https-3A__chrome.google.com_webstore_detail_axe_lhdoppojpmngadmnindnejefpokejbdd-3Fhl-3Den-2DUS&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=1dXTpkaxcJAWFHXRulvC3AXNO7ZVwl0gYUvi1fz6Kqs&s=y6aME0KRkEtR-xcdSNl15Tj6dAqOgnwhF73KFo3aI0c&e=>

Life is ten percent what happens to you and ninety percent how you respond to it. - Lou Holtz
Received on Friday, 15 July 2016 19:28:28 UTC