W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2016

Re: When it is appropriate to use wpt...

From: Andreas Tolfsen <ato@mozilla.com>
Date: Thu, 21 Apr 2016 19:54:59 +0100
Message-ID: <CAL_dnaV1RS8opSrbDnGUJY+Gqs90cYzvUrXHJx6M=m7mteLSPQ@mail.gmail.com>
To: Geoffrey Sneddon <me@gsnedders.com>
Cc: Shane McCarron <shane@spec-ops.io>, public-test-infra <public-test-infra@w3.org>, jyasskin@google.com, Simon Stewart <simon.m.stewart@gmail.com>

On 21 April 2016 at 19:39, Geoffrey Sneddon <me@gsnedders.com> wrote:
> On Thu, Apr 21, 2016 at 6:06 PM, Shane McCarron <shane@spec-ops.io> wrote:
>> My intent all along was to do this work within the WPT structure.  However,
>> as I discussed this in #testing today it became clear to me that there might
>> be an architectural problem with doing so.  Consequently, before I start
>> down a path that you, the W3C testing gods, might object to, I thought I
>> would ask for your blessing (or at least informed apathy).
>> There is a class of specifications that *can* be exercised from a user
>> agent, but which have dependencies OUTSIDE of the user agent - and
>> potentially outside of the test "server".  For example, while Web Payments
>> has a "browser-payment-api" component, it has other pieces in its
>> architecture that will have conformance criteria and need to be tested (e.g.
>> a "payment app" that is a remote web service to which messages are passed
>> from the user agent). Tests for these pieces can be driven by a user agent.
>> They could also be driven by some command line test driver.

Providing said server as a service the wptrunner test harness spins up
seems reasonable.  We do this for testing SSL and WebSockets.

> I remember discussing with Jeffrey Yasskin what was needed for Web
> Bluetooth, which has arguably similar needs to Web Payments. While
> that doesn't have a server component that needs testing, it does have
> a similar problem insofar it has a third-party API that needs dealt
> with (though a Bluetooth device rather than a payment app). I think
> it'd be nice to come up with a solution that addresses both needs.
> Both need some form of pluggability that real implementations may not
> have (esp. in the Bluetooth case!), given I presume on iOS for example
> Apple Pay will be the option.
> I wonder if a WebDriver API would make sense (WebDriver API use
> inevitably already has security consequences, so I don't think it's
> too unreasonable); thoughts, anyone, or any better ideas?

WebDriver provides a way to get privileged access to browser
internals, as it effectively bypasses traditional content security
permissions, as Geoffrey is saying.

At TPAC in Sapporo we had a meeting with the WebBluetooth WG about the
testability aspect of their specification, and found that a reasonable
extension to WebDriver would be a permissions API. There appears to be
interest for this both from Google and the Mozilla Servo team.

Right now we’re focusing on finishing the “base level” WebDriver
specification, but it is the intention of the Browser Tools- and
Testing WG to continue work on a permissions API extension soon after
it is pushed to Rec.

In practical terms we plan to introduce a “meta circular” in-content
JS client to access the WebDriver session that wptrunner runs the
tests in. The work on this is described in

This could allow a developer to write tests such as
webdriver.click(element).then(() => assert_false(element.disabled) in
order to test trusted click events.  We could have a similar
cross-browser API for accepting and dismissing permission dialogues.
Simon (Cc’ed) and I discussed one in Japan.

>> If we have a general case mechanism (e.g., validating JSON data via JSON
>> Schema and Python) is there a process for getting that general mechanism
>> into the core framework?
> I think adding them to wpt-tools would be reasonable.
> I know James Graham feels strongly about keeping tests self-contained
> and in one file as much as possible, and while I remember him speaking
> a fair bit when we popped into the Web Payments meeting at TPAC, I
> can't remember what was said. I expect he'll have strong opinions
> about how to do this. :)

Self-containment I believe is an absolute requirement from Mozilla’s
side, as the browser we are testing in CI is instructed to crash on an
assertion when it attempts to access remote resources. We do this for
hermetical purposes so that tests are reproducible.
Received on Thursday, 21 April 2016 18:55:48 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:12 UTC