Re: Testing WebDriver in Web Platform Tests

Yesterday in the #webdriver IRC channel, Andreas requested some 
clarification:

 > Please share your concerns with using the wdclient library. Your 
email that
 > was forwarded seemed to mostly focus on test harness features, which 
wdclient
 > isn't responsible for.

My concern about the use of any library (not wdclient in particular) is 
that it
increases the complexity of the code under test. This can make the tests 
harder
to understand overall, as critical details are abstracted away by a 
convenient
API. I prefer to use a binding library when testing applications because in
those contexts, I'm asserting the behavior of business logic, and 
details about
the WebDriver protocol are irrelevant.

Here though, the WebDriver protocol is itself the test subject. We should
expect tests for (for example) "Add Cookie" to contain an object definition
like:

     { "cookie": { "name": "foo", "value": "bar" } }

...rather than either of the two phrasings currently allowed [1]:

     cookies["foo"] = "bar"

     class Wrapper:
       def __init__(self, value):
         self.value = value

     cookies["foo"] = Wrapper("bar")

Beyond obscuring the behavior under test, that kind of convenience is itself
susceptible to bugs (e.g. note that the HTTP request produced by the 
previous
example is incorrect in the current implementation) and requires additional
maintenance.

In the current project structuring, maintenance is a bit of a burden because
fixing a test bug involves submitting a patch to one project and then 
updating
a submodule. In this way, the "web platform tests" project is not the 
"single
source of truth" for the protocol--the expected behavior is spread across
multiple locations. (I'll admit that it's only my presumption that 
having the
tests act as "single source of truth" is actually desirable for the
specification maintainers.)

I'm the new guy here, though, and I don't mean to rock the boat too much. If
folks see value in these concerns, then we may be able to address them while
still leveraging the library. Here's one possible scenario:

- Move the library into the "Web Platform Tests" project
- Move the pytest "fixture" definitions from the "wptrunner" project to the
   "Web Platform Tests" project
- Formally define a policy in `webdriver/README.md` which states that the
   command under test must not be issued by the library code (meaning 
library
   use is limited to streamlining extraneous details)

This isn't optimal because it requires contributors/maintainers to
discover/understand/conform to an informal policy, but it could satisfy a
majority of my concerns. What would you all think?

Mike

[1] 
https://github.com/w3c/wdclient/blob/1e751092c7f8f11748198fcb0c8c542f1216b466/webdriver/client.py#L142-L145

Received on Wednesday, 25 January 2017 18:13:32 UTC