When it is appropriate to use wpt...

Hi!

This is my first message to this list, so please be gentle.

I volunteered to act as Test Wrangler for the Web Payments activity [1].
Pity me.

This group has just now published its first public working drafts, and it
seems a good time to get the test writing started!

My intent all along was to do this work within the WPT structure.  However,
as I discussed this in #testing today it became clear to me that there
might be an architectural problem with doing so.  Consequently, before I
start down a path that you, the W3C testing gods, might object to, I
thought I would ask for your blessing (or at least informed apathy).

There is a class of specifications that *can* be exercised from a user
agent, but which have dependencies OUTSIDE of the user agent - and
potentially outside of the test "server".  For example, while Web Payments
has a "browser-payment-api" component, it has other pieces in its
architecture that will have conformance criteria and need to be tested
(e.g. a "payment app" that is a remote web service to which messages are
passed from the user agent). Tests for these pieces can be driven by a user
agent.  They could also be driven by some command line test driver.

Another example is Web Annotation - it has a data model, a vocabulary, and
a protocol.  It has no browser-specific requirements and no defined UI, but
implementations can (and are) browser based on the client side so it is
appropriate to exercise those inside a user agent.  There are also
implementation pieces that are on a "annotation server" somewhere.  Those
pieces could be exercised from a user agent, but the real meat of the test
is in the dialog between the user agent and the server - the conformance
requirements are on the REMOTE end - if that makes any sense.

So, here are my questions:

   1. Is it appropriate to incorporate tests and tools into WPT that will
   potentially utilize and/or exercise resources outside of the test server
   (e.g., pulling in information from a tester-defined remote resource or
   testing the conformance of a tester-defined remote resource)?
   2. Are extensions to WPT limited to using Python (2.7) and github?  Are
   there other tools (e.g., Ruby, Node) that can be used by extensions?
   3. If we have a general case mechanism (e.g., validating JSON data via
   JSON Schema and Python) is there a process for getting that general
   mechanism into the core framework?


I am sure that I will have other questions... Sorry!

[1] w3.org/Payments/WG

-- 
Shane McCarron
Projects Manager, Spec-Ops