Refactoring and completing tests for user prompt behavior

Hi all,

If you lately followed the web-platform-tests repository for wdspec test
changes, you might have noticed that I started a kinda large refactoring
of the existent user prompt tests. This is an ongoing effort on my side
with the goal to have a complete set of those tests for all commands in
the near future.

The improvements as already landed are:

* Having a custom marker `capabilities` which allows you to specify the
required capabilities for a test. There is no need anymore for the
complicated usage of the `new_session` fixture, which now is even only
available for the New Session tests.

* Only create a new session (and restart the browser) if the requested
capabilities are different from those of the previous test.

* Removed around 66% of code from user prompt tests by using
parametrization for `alert`, `confirm`, and `prompt` dialogs.

* Added checks to ensure that the dialog was correctly closed (accept/ok
vs. dismiss/cancel)

* Added checks to ensure that the underlying command is run for the
`accept` and `dismiss` prompt behavior, and not run for the others.

Also there are some things left to do, and which will land during the
next days:

* If possible even further reduction of code by more parametrization.

* Finishing off all existing user prompt tests for missing subtests -
currently marked as TODO in the test files.

* Adding user prompt tests for all commands which don't have those tests
yet; which will be ** 17 ** beside the ** 19 ** we currently have covered!


Please note that when all the above is done the duration of running the
tests will largely increase. A big factor for us is at least the restart
of Firefox for each test which has a different `unhandledPromptBehavior`
capability set. I want to give this information ahead of time in case of
your CI system has to be changed.

A way to make those tests faster would be to not have to restart the
browser in case of session deletion / creation. We started to
investigate this for us on
https://bugzilla.mozilla.org/show_bug.cgi?id=1470120.

We aren't sure yet if that is possible, or even wanted. If you have
ideas please let us know on the bug, or via a reply to this message. Thanks.

In case of questions about this topic please let me know.

Thanks

-- 
Henrik Skupin
Senior Software Engineer
Mozilla Corporation

Received on Wednesday, 1 August 2018 08:30:18 UTC