Re: When it is appropriate to use wpt...

Disclaimer: I haven't read what was discussed in #testing. I also
don't have too much clue about Web Payments.

On Thu, Apr 21, 2016 at 6:06 PM, Shane McCarron <shane@spec-ops.io> wrote:
> Hi!
>
> This is my first message to this list, so please be gentle.
>
> I volunteered to act as Test Wrangler for the Web Payments activity [1].
> Pity me.
>
> This group has just now published its first public working drafts, and it
> seems a good time to get the test writing started!
>
> My intent all along was to do this work within the WPT structure.  However,
> as I discussed this in #testing today it became clear to me that there might
> be an architectural problem with doing so.  Consequently, before I start
> down a path that you, the W3C testing gods, might object to, I thought I
> would ask for your blessing (or at least informed apathy).
>
> There is a class of specifications that *can* be exercised from a user
> agent, but which have dependencies OUTSIDE of the user agent - and
> potentially outside of the test "server".  For example, while Web Payments
> has a "browser-payment-api" component, it has other pieces in its
> architecture that will have conformance criteria and need to be tested (e.g.
> a "payment app" that is a remote web service to which messages are passed
> from the user agent). Tests for these pieces can be driven by a user agent.
> They could also be driven by some command line test driver.

I remember discussing with Jeffrey Yasskin what was needed for Web
Bluetooth, which has arguably similar needs to Web Payments. While
that doesn't have a server component that needs testing, it does have
a similar problem insofar it has a third-party API that needs dealt
with (though a Bluetooth device rather than a payment app). I think
it'd be nice to come up with a solution that addresses both needs.
Both need some form of pluggability that real implementations may not
have (esp. in the Bluetooth case!), given I presume on iOS for example
Apple Pay will be the option.

I wonder if a WebDriver API would make sense (WebDriver API use
inevitably already has security consequences, so I don't think it's
too unreasonable); thoughts, anyone, or any better ideas?

> Another example is Web Annotation - it has a data model, a vocabulary, and a
> protocol.  It has no browser-specific requirements and no defined UI, but
> implementations can (and are) browser based on the client side so it is
> appropriate to exercise those inside a user agent.  There are also
> implementation pieces that are on a "annotation server" somewhere.  Those
> pieces could be exercised from a user agent, but the real meat of the test
> is in the dialog between the user agent and the server - the conformance
> requirements are on the REMOTE end - if that makes any sense.
>
> So, here are my questions:
>
> Is it appropriate to incorporate tests and tools into WPT that will
> potentially utilize and/or exercise resources outside of the test server
> (e.g., pulling in information from a tester-defined remote resource or
> testing the conformance of a tester-defined remote resource)?

Yes, I think it is, though I think we should add tools to wpt-tools to
keep everything self-contained.

> Are extensions to WPT limited to using Python (2.7) and github?  Are there
> other tools (e.g., Ruby, Node) that can be used by extensions?

I'd rather we used Python for the sake of being consistent with what's
already required.

> If we have a general case mechanism (e.g., validating JSON data via JSON
> Schema and Python) is there a process for getting that general mechanism
> into the core framework?

I think adding them to wpt-tools would be reasonable.

I know James Graham feels strongly about keeping tests self-contained
and in one file as much as possible, and while I remember him speaking
a fair bit when we popped into the Web Payments meeting at TPAC, I
can't remember what was said. I expect he'll have strong opinions
about how to do this. :)

/Geoffrey

Received on Thursday, 21 April 2016 18:39:29 UTC