Re: Testing WebDriver in Web Platform Tests

Mike Pennisi <mike@bocoup.com> writes:

> My concern about the use of any library (not wdclient in particular)
> is that it increases the complexity of the code under test. This can
> make the tests harder to understand overall, as critical details are
> abstracted away by a convenient API.

In general I agree with this sentiment.  But there is also a point to
be made that a loosely-coupled local end-like library can make it less
challenging and tiresome to write tests that involve interaction with
other commands that are not under test.

When I use the term “local end-like” I mean that wdclient is not your
average WebDriver client you will find in the wild.  There are many
conveniences it does not provide, with good reason.

> I prefer to use a binding library when testing applications because
> in those contexts, I'm asserting the behavior of business logic, and 
> details about the WebDriver protocol are irrelevant.
> 
> Here though, the WebDriver protocol is itself the test subject. We
> should expect tests for (for example) "Add Cookie" to contain an
> object definition like:
> 
>      { "cookie": { "name": "foo", "value": "bar" } }
> 
> ...rather than either of the two phrasings currently allowed [1]:
> 
>      cookies["foo"] = "bar"
> 
>      class Wrapper:
>        def __init__(self, value):
>          self.value = value
> 
>      cookies["foo"] = Wrapper("bar")

If I were testing cookies, I would assert that the endpoint behaved
correctly, e.g. that it was returning the correct HTTP status code and
that the response body was parsable as JSON.  I would test that the
body contained the expected fields and that the fields’ values had the
correct types.  Then I would check invalid data input and types to see
if the expected errors are returned, and finally I would continue with
bounds checks of the values.

Only after testing all these things, would I consider using an
abstraction API to run a set of data tests to check that the API is
behaving as expected.

> Beyond obscuring the behavior under test, that kind of convenience is
> itself susceptible to bugs […] and requires additional maintenance.

There is always a chance a bug would be hidden in the abstraction
unless you have a good set of protocol-level tests that assert the
endpoint, input arguments, and sanity of the response.

As I showed in my last email, it is possible to test all these things
using the `http` fixture to send malformed data and the
`session.send_command` to send raw data structures to arbitrary
endpoints.

It is true that we _are_ trading away some specificity for convenience
when abstracting away details, but the bigger question here is if this
is an acceptable trade-off as opposed to writing out every JSON literal
and its expected response?

> In the current project structuring, maintenance is a bit of a burden
> because fixing a test bug involves submitting a patch to one project
> and then updating a submodule. In this way, the "web platform tests"
> project is not the "single source of truth" for the protocol--the
> expected behavior is spread across multiple locations.

We don’t have this problem at Mozilla since all the source code is
checked out in a single tree, letting us make all the changes at once.

However, I’ve felt this pain before and I agree the situation is not
great.

> - Move the library into the "Web Platform Tests" project

I don’t think there is an easy way to do this right now.  wdclient is a
submodule of wpt-tools, which again is a submodule of web-platform-
tests.

You might have to make some changes to wptrunner for it to know where
to pick up wdclient from.

> - Move the pytest "fixture" definitions from the "wptrunner" project
> to the "Web Platform Tests" project

The `session` fixture reuses certain resources that are only available
to wptrunner, so this cannot be moved.

But it is possible to create config.py file (or some other name I’ve
forgotten) that pytest will pick up automatically, so moving the
remaining fixtures into the /webdriver folder makes sense.

> - Formally define a policy in `webdriver/README.md` which states that
> the command under test must not be issued by the library code
> (meaning library use is limited to streamlining extraneous details)

I don’t understand what problem you think you are solving with policy.

Each command needs to be adequately tested, which means all the formal
aspects of a command’s input and output must be asserted.  This
involves type checks, status codes, bounds checks, data coherence, &c. 
If a test does not conform to these standards, it should be rejected in
review.

Received on Thursday, 26 January 2017 14:53:48 UTC