Re: Testing WebDriver in Web Platform Tests

Hi Mike and Sam,

Sam Uong <samuong@google.com> writes:

> My immediate thought is that we want to avoid a few things:
> code duplication - particularly for more complex parameters, such as
> those described in the Actions section of the spec

I’ve just r+’ed a series of patches that implements a high-level API
for actions in wdclient.  These changes will be shortly be upstreamed
to WPT.

> injecting errors - how easy will it be to test error conditions by
> sending malformed requests?

wdclient exposes a primitive to send invalid command bodies:

 session.send_command("POST", "actions", {"obviouslyIncorrect": 42})

The WPT harness also provides an ‘http’ fixture for doing more complex
interaction with the remote end:

 with http.get("/session/%s/window/size" % session.session_id) as resp:
     assert resp.status == 200
     body = json.load(resp)
 assert "width" in body
 assert "height" in body
 assert isinstance(body["width"], int)
 assert isinstance(body["height"], int)

Both the ‘send_command’ method and the ‘http’ fixture lets you test
calls to invalid endpoints and malformed input to commands.

> On Tue, Jan 24, 2017 at 10:24 AM Mike Pennisi <mike@bocoup.com>
> wrote:
> 
> > Specifically: would anyone object to my not using the "wdclient"
> > library [2]? We definitely need to take steps to lessen
> > duplication, but I'd much prefer to use traditional unit testing
> > patterns for this--"setup/teardown" routines, helper functions, or
> > even pytest's "fixture" feature. This will make the
> > tests simpler and more direct, and that promotes readability (which
> > is good for "drive-by" contributors), traceability (which is good
> > for implementers wondering why a test fails), and specification
> > parity (which is good for test suite maintainers trying to re-
> > assess coverage of a given section of the spec).

The wdclient library was written with the explicit intent to be used in
the WPT tests for WebDriver.  It exposes more primitive functions to
enable testing of the WebDriver protocol than usual WebDriver clients,
and the features we are lacking we can add.

wdclient is itself not a test harness.  The small, but existing body of
WebDriver tests in WPT uses pytest as harness:

 https://github.com/w3c/web-platform-tests/blob/master/webdriver

pytest is integrated in wptrunner, which is the test runner, so that
browser vendors can more easily integrate them to their existing
continuous integration environments.  pytest gives us so called
“fixtures” which I have found lends themselves very well to the
functional style of testing we are looking to do for WebDriver.

You can read more about fixtures here:

 http://doc.pytest.org/en/latest/fixture.html

When I wrote the integration with wptrunner in May last year, I also
wrote a blog post with some more details and usage examples.  Even
though the blog post is a bit out of date, it may still be useful:

 https://sny.no/2016/05/wdspec

In particular, due to the way we integrate wptrunner into the Mozilla
all-purpose command tool ‘mach’ (German for “do”), running WPT tests
from mozilla-central is very easy:

 % ./mach build
 …
 % ./mach wpt TESTFILE

Due to some unfortunate and hopefully temporary issues, you currently
need to have geckodriver compiled and available on the system to run
the WebDriver tests:

 % ./mach build
 …
 % ./mach wpt --webdriver-binary /path/to/geckodriver TESTFILE

> > I think we need to prioritize these qualities over the developer
> > ergonomics provided by a binding library.

I’m all ears if you have an idea how we could do this more effectively
and more ergonomically, but let me just reiterate that it’s not
wdclient’s foremost design goal to be an end-user WebDriver client
library: its purpose is to provide us with the necessary primitives to
economically write WPT tests specifically for the WebDriver standard,
and we are free to change it to suit our needs.

Received on Tuesday, 24 January 2017 20:55:53 UTC