Re: Request for bikeshedding

On 27/02/14 02:17, Wilhelm Joys Andersen wrote:

>
> The HTML files contain human-readable steps and pass conditions, and
> can be run and understood independently of the WebDriver script. This
> is required both to be able to run the tests manually in browsers
> without WebDriver support, and to assist browser vendor developers
> fixing bugs triggered by a particular test case.

Yes, this is a requirement.

> The WebDriver script loads each of the test files in sequence,
> performing steps identical to those a human would, checking for the
> same assertions.

> We need to make both setup and authoring of WebDriver tests anywhere
> in the web-platform-test repo as simple as possible. Ideally before we
> start pushing our conventions onto other groups. Hence this request
> for bikeshedding. What sources of friction can you see in the proposed
> format?

Webdriver tests need to have some unique way of being identified by the 
manifest generator so that the complete list of webdriver tests can be 
retrieved. At the moment manual tests use a -manual suffix. I suggest we 
do something similar with webdriver.py files e.g. test-foo-webdriver.py 
(better names welcome).

It is very unclear to me how the browser launching is supposed to work. 
Presumably webdriver wants to launch its own browser? In automation this 
doesn't seem ideal because we want quite fine control over the browser 
process with a specially configured profile, etc. We also need to be 
able to tell when the browser crashed and handle that in a good way, and 
to ensure that both the webdriver process and the browser process get 
killed if a test takes too long. I think from that point of view a model 
where we could hand the test script a port number to connect to would be 
preferable for automation scenarios.

I personally dislike unittest and would prefer a model where you wrote a 
test like:

def test_foo(test, browser):
     blah = browser.find_element("spam")
     test.assert_equals(blah, whatever)

> On my own list of issues are:
>
>   * We are currently not using wptserve. This is being worked on.

Please ask if you need help here.

>   * We don't pull in all the APIs we might need. I had to manually pull
> in ActionChains.
>
>   * The required files to run WebDriver tests are essentially part of
> the WebDriver conformance test suite, not a shared resource between
> all test suites in web-platform-test.
>
>   * Referencing self.driver on every second line gives me writer's
> cramp. We should alias this to something shorter in our helper files.
>
>   * The same applies to the find methods. 99% of all tests in
> web-platform-test will probably use find_element_by_css_selector. It
> would be great to alias this to something shorter, as a poor man's
> page object. b.find('#result') would be nice.

This seems like something wrong at the Python WebDriver API level rather 
than with specific fixes for wpt?

>   * Writing test names like test_onclick_handler_on_button is suboptimal.

Suboptimal in what sense? It's annoyingly long, but might be easier than 
writing separate test titles in e.g. docstrings.

> What else?
>
> And which of the above are fixable - and desirable to fix? :)
>

Received on Monday, 3 March 2014 17:49:37 UTC