Re: When it is appropriate to use wpt...

Thanks for your reply - some comments:

On Thu, Apr 21, 2016 at 1:39 PM, Geoffrey Sneddon <me@gsnedders.com> wrote:

> Disclaimer: I haven't read what was discussed in #testing. I also
> don't have too much clue about Web Payments.
>
> On Thu, Apr 21, 2016 at 6:06 PM, Shane McCarron <shane@spec-ops.io> wrote:
> > Hi!
> >
> > This is my first message to this list, so please be gentle.
> >
> > I volunteered to act as Test Wrangler for the Web Payments activity [1].
> > Pity me.
> >
> > This group has just now published its first public working drafts, and it
> > seems a good time to get the test writing started!
> >
> > My intent all along was to do this work within the WPT structure.
> However,
> > as I discussed this in #testing today it became clear to me that there
> might
> > be an architectural problem with doing so.  Consequently, before I start
> > down a path that you, the W3C testing gods, might object to, I thought I
> > would ask for your blessing (or at least informed apathy).
> >
> > There is a class of specifications that *can* be exercised from a user
> > agent, but which have dependencies OUTSIDE of the user agent - and
> > potentially outside of the test "server".  For example, while Web
> Payments
> > has a "browser-payment-api" component, it has other pieces in its
> > architecture that will have conformance criteria and need to be tested
> (e.g.
> > a "payment app" that is a remote web service to which messages are passed
> > from the user agent). Tests for these pieces can be driven by a user
> agent.
> > They could also be driven by some command line test driver.
>
> I remember discussing with Jeffrey Yasskin what was needed for Web
> Bluetooth, which has arguably similar needs to Web Payments. While
> that doesn't have a server component that needs testing, it does have
> a similar problem insofar it has a third-party API that needs dealt
> with (though a Bluetooth device rather than a payment app). I think
> it'd be nice to come up with a solution that addresses both needs.
> Both need some form of pluggability that real implementations may not
> have (esp. in the Bluetooth case!), given I presume on iOS for example
> Apple Pay will be the option.
>

I suspect it will be an option, surely.  The specifications allow for
multiple payment apps to register their availability with the platform.


>
> I wonder if a WebDriver API would make sense (WebDriver API use
> inevitably already has security consequences, so I don't think it's
> too unreasonable); thoughts, anyone, or any better ideas?
>

I have a view of this where vendors could supply their own web driver based
"scripts" to drive the parts of a transaction that might require human
intervention.  There is a similar problem with Web Annotation - where there
is no defined UI per se, but of course each client will have one and there
needs to be a way to prod that client until they spit out an annotation
object.


>
>
> > Is it appropriate to incorporate tests and tools into WPT that will
> > potentially utilize and/or exercise resources outside of the test server
> > (e.g., pulling in information from a tester-defined remote resource or
> > testing the conformance of a tester-defined remote resource)?
>
> Yes, I think it is, though I think we should add tools to wpt-tools to
> keep everything self-contained.
>

Can you expand on this?  My examples of external resources would be things
like the Visa(tm) Web Payment service.  That's not something we can put
into a self-contained environment.  Maybe *they* can, and I suppose we
could require that.  But it feels like a high bar to make them leap over.
Also, if the Visa(tm) Web Payment service is what is supposed to be under
test, and we test something different, aren't we violating the basic
principle of testing ("If it's not the same, it's different.")?


>
> > Are extensions to WPT limited to using Python (2.7) and github?  Are
> there
> > other tools (e.g., Ruby, Node) that can be used by extensions?
>
> I'd rather we used Python for the sake of being consistent with what's
> already required.
>

Understood.


>
> > If we have a general case mechanism (e.g., validating JSON data via JSON
> > Schema and Python) is there a process for getting that general mechanism
> > into the core framework?
>
> I think adding them to wpt-tools would be reasonable.
>
> I know James Graham feels strongly about keeping tests self-contained
> and in one file as much as possible, and while I remember him speaking
> a fair bit when we popped into the Web Payments meeting at TPAC, I
> can't remember what was said. I expect he'll have strong opinions
> about how to do this. :)
>

I am a strong proponent of this approach.  To the point that I often use
"test" files that contain information that gets parsed out by the tools
that represent different ends of a transaction.  That sort of an
architecture is relatively easy to put in place when there is as much
control as we seem to have in the wpt environment.


-- 
Shane McCarron
Projects Manager, Spec-Ops