W3C home > Mailing lists > Public > public-test-infra@w3.org > July to September 2016

Re: Communicating between the WPT test window and a separate application on the client

From: Dylan Barrell <dylan.barrell@deque.com>
Date: Fri, 15 Jul 2016 14:49:14 -0400
Message-ID: <CAKGs1QZySo2dm8-K3zB-MDuQockuNuaCrHotF=oqAWuy4U025w@mail.gmail.com>
To: "Gunderson, Jon R" <jongund@illinois.edu>
Cc: Shane McCarron <shane@spec-ops.io>, David Brett <dbrett@microsoft.com>, "public-aria-test@w3.org" <public-aria-test@w3.org>, public-test-infra <public-test-infra@w3.org>
That spec is orthogonal to the questions I raised.

--Dylan


On Fri, Jul 15, 2016 at 2:47 PM, Gunderson, Jon R <jongund@illinois.edu>
wrote:

> Dylan,
>
> The test pages will be written in HTML, CSS and Javascript, a JSON object
> int he page will be used to identify platform specific mapping informaiton.
>
> Here is a link to  a DRAFT of proposed format for a typical test page:
>
> <https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Draft_ARIA_Test_Case_Format>
> https://www.w3.org/wiki/ARIA_1.1_Automated_Testing#Draft_ARIA_Test_Case_Format
>
>
> Jon
>
>
> From: Dylan Barrell <dylan.barrell@deque.com>
> Date: Friday, July 15, 2016 at 1:34 PM
> To: Shane McCarron <shane@spec-ops.io>
> Cc: David Brett <dbrett@microsoft.com>, "public-aria-test@w3.org ARIA
> Implementation Testing" <public-aria-test@w3.org>, Jon Gunderson <
> jongund@illinois.edu>, public-test-infra <public-test-infra@w3.org>
> Subject: Re: Communicating between the WPT test window and a separate
> application on the client
>
> I would like some clarifying information:
>
> 1) Do you have an opinion on what language the tests will be written in?
> Would it be a requirement to have this be bindable to any language or do we
> just propose picking one?
> 2) In your diagrams, there is some control instance running the browsers.
> In the Web testing world, the communication would normally be along the
> lines of
>
> TEST RUNNER <-----------> BROWSER AUTOMATION (e.g. WebDriver)
> <-----------> BROWSER
>
> How does this picture map to the picture(s) you drew above? Could it be
> seen as follows
>                            <-----------> BROWSER AUTOMATION (e.g.
> WebDriver) <-----------> BROWSER
> TEST RUNNER
>                            <-----------> LOCAL AT SHIM <-----------> OS
>
> Where the test runner is responsible for communicating with both the
> browser automation AND the AT shim?
>
> If this is the case, then the logic for executing individual or batch
> tests is inside the test runner and could be seen as essentially an
> expansion of the WebDriver API. If this is the case, it could be bound to
> any language including JS (Node), Java, etc.
>
> --Dylan
>
>
> On Fri, Jul 15, 2016 at 1:50 PM, Shane McCarron <shane@spec-ops.io> wrote:
>
>> I'm out of position right now. I understand what you are proposing, but I
>> think it flies in the face of the general architecture of tests under WPT.
>> Basically in the w3c you need to have individual tests for each feature and
>> those should be individually executable.  I can imagine how to accomplish
>> that in your model... Let me ponder a little more.
>>
>> On Jul 15, 2016 12:40 PM, "David Brett" <dbrett@microsoft.com> wrote:
>>
>> I’d like to suggest another architecture that could be simpler. Instead
>> of creating a line of communication between the AT and the main browser
>> window for each test, we could instead pass the entire list of files (and
>> requirements) at once, let the same code that manages the AT handle running
>> through the tests, and then pass all the data back at the end. Here is my
>> own (terrible ASCII art) diagram:
>>
>>
>>
>> WEB BROWSER
>>
>>  MAIN WINDOW
>>
>>      ^
>>
>>      |
>>
>>    MAGIC
>>
>>      |
>>
>>      v                     CHILD WINDOW
>>
>> LOCAL AT SHIM  <----->  FOR INDIVIDUAL TESTS
>>
>>
>>
>> Obviously we have to make that “magic” connection only twice instead of
>> hundreds of times which will most likely be more reliable and save time. I
>> don’t think this will result in any extra work if we use A11y, since the
>> system is already set up to handle a list of tests.
>>
>>
>>
>> As far as how that communication works, I think it would be pretty
>> straightforward to include the test requirement files in a directory of the
>> WPT repo and let A11y iterate over those. Sending the results through a
>> websocket once the tests are completed shouldn’t be too hard either.
>>
>>
>>
>> *From:* Gunderson, Jon R [mailto:jongund@illinois.edu]
>> *Sent:* Friday, July 15, 2016 7:30 AM
>> *To:* Shane McCarron <shane@spec-ops.io>; public-test-infra <
>> public-test-infra@w3.org>; public-aria-test@w3.org
>> *Subject:* RE: Communicating between the WPT test window and a separate
>> application on the client
>>
>>
>>
>> Shane thank you again for getting this conversation started and your
>> interest in getting ARIA testing as part of WPT.
>>
>>
>>
>> Just for clarification.  The “fake AT” is basically a application that
>> exposes information about a platform specific accessibility API as WPT or
>> some other tool exercises a ARIA test page.   So the application needs to
>> look the accessibility tree and related events and get that information
>> back into the browser for comparison with expected results for us to
>> utilize the WPT framework to its fullest extent.
>>
>>
>>
>> Jon
>>
>>
>>
>>
>>
>> *From:* Shane McCarron [mailto:shane@spec-ops.io <shane@spec-ops.io>]
>> *Sent:* Friday, July 15, 2016 8:43 AM
>> *To:* public-test-infra <public-test-infra@w3.org>;
>> public-aria-test@w3.org
>> *Subject:* Communicating between the WPT test window and a separate
>> application on the client
>>
>>
>>
>> Hi!
>>
>>
>>
>> The ARIA Working Group is investigating various ways to automate the
>> testing of ARIA - which requires testing the accessibility api and its
>> communication with assistive technologies (AT) on a bunch of platforms.
>> Obviously, this is a bit of a challenge.  The current thinking is that a
>> fake AT can be provided on each platform.  The fake AT is started by the
>> tester (or test automation environment) prior to starting a test run.  Once
>> it is running and has found the test window, it will capture the
>> accessibility tree and events as the tests set up and manipulate the DOM.
>> Simple enough.
>>
>>
>>
>> Except, of course, for getting the information from the fake AT back into
>> the test window.  Consider the following (terrible ASCII art) diagram:
>>
>>
>>
>>  WEB BROWSER   <----->  CHILD WINDOW
>>
>>  MAIN WINDOW            FOR INDIVIDUAL TEST
>>
>>                               ^
>>
>>                               |
>>
>>                             MAGIC
>>
>>                               |
>>
>>                               v
>>
>>                          LOCAL AT SHIM
>>
>>
>>
>> The "MAGIC" is where I am playing right now.  Here are some of my ideas:
>>
>>    1. Shim is passed a URI on the WPT server on startup (or finds the
>>    URI when it finds the test window). Communicates with it through a
>>    websocket, and the window in which the test is running communicates with
>>    the same websocket endpoint.  Data is relayed that way.  This seems the
>>    most portable to me.
>>    2. Shim runs a simple HTTP listener.  Child window communicates with
>>    that using HTTP (websocket or simple HTTP GET) on localhost.  This requires
>>    implementing a messaging stack... which doesn't feel very easy on every
>>    platform but is probably do-able.  It might also violate CORS stuff, but
>>    again - do-able.
>>    3. Rely on some sort of native messaging that is platform specific.
>>    This doesn't feel scalable to me.  It would also mean modifying the WPT
>>    part of this any time we wanted to add another platform that had a
>>    different native messaging capability.
>>    4. Use a ServiceWorker in some magical way that I probably don't
>>    understand.  Feels like a steep learning curve.  Also, they don't seem to
>>    be widely supported yet.
>>
>> My hope is that some of you have already thought about a similar problem
>> (there is a native application running on the platform under test that
>> needs to send messages into the test in order to evaluate success or
>> failure).  So... any ideas?
>>
>> --
>>
>> Shane McCarron
>>
>> Projects Manager, Spec-Ops
>>
>>
>>
>
>
> --
> Download the aXe browser extension for free:
>
> Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__addons.mozilla.org_en-2DUS_firefox_addon_axe-2Ddevtools&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=_JVA5omJG9Ir99XWLuJRR0feRTWZKLYVYxlNXWGn_pA&s=Labt5u56v6mvY5zOGP3drcLE1gWLoTNMGcfTGH4QbvE&e=>
> Chrome:
> https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__chrome.google.com_webstore_detail_axe_lhdoppojpmngadmnindnejefpokejbdd-3Fhl-3Den-2DUS&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=REZD8fc2AwufInstfW3L5jSLVS8bjZtAodDOhat7yAI&m=_JVA5omJG9Ir99XWLuJRR0feRTWZKLYVYxlNXWGn_pA&s=VeYI_KeRTXk6sXG4Vt24RShA1lVJ7T2tZMHqCwPdJho&e=>
>
> Life is ten percent what happens to you and ninety percent how you respond
> to it. - Lou Holtz
>
>


-- 
Download the aXe browser extension for free:

Firefox: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools
Chrome:
https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US

Life is ten percent what happens to you and ninety percent how you respond
to it. - Lou Holtz
Received on Friday, 15 July 2016 18:49:45 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:12 UTC