Re: Plan for reftests

On 26/01/15 17:19, James Graham wrote:
> In order to better align with what CSS do and make reftests more
> flexible, I plan to implement some changes to the way that reftests work
> in the future:
> 
> * Reftests will be identified as files that are not other kinds of tests
> and contain <link rel=match> or <link rel=mismatch>.
> 
> * Multiple such <link> elements in a file will have OR semantics i.e. in
> order for the test to pass it much match one of the two conditions. This
> is expected to be rare but is sometimes needed where specs leave things
> underdefined.
> 
> * In order to get AND semantics, refs may themselves <link> to other refs.
> 
> * In order to distinguish a file that should be considered a test from
> one that should be considered a pure reference, any reference containing
> <link rel=[mis]match> elements must have -ref or -notref at the end of
> the filename, before the extension.
> 
> * Reference cycles will be permitted e.g. A == B == C == B. It is not
> quite clear what to do about cycles including the test itself; in the ==
> case these seem likely to be bugs e.g A == A or A == B == A. However
> there are != cases where this makes sense e.g. A == B != C != A.
> 
> * When executing, the graph formed with the resources as vertexes and
> edges as <link>s will be flattened into a tree by breaking cycles. This
> tree will be processed in a depth-first search with the goal of making
> it to a leaf vertex along only edges where the comparison succeeds. If
> such a leaf is found the comparison terminates early. If Something Bad
> happens before such an edge is reached (e.g. a browser crash or an error
> of some kind) the test will terminate with that status.
> 
> Does that make sense?
> 

This has now landed. Any wpt reftest should now also be a valid CSS
reftest; the converse is not yet true because they still allow _ref and
/references/ filenames which we (intentionally) don't support.

In the same batch of changes a few other things landed:

* [wptrunner] Reftest support for Chrome (and, in the future, anything
else we can support via Selenium / WebDriver).

* [wptrunner] Some integration selftests that can be run when developing
changes. The process for running them is simply "python test.py" in the
tests directory.

* [wptrunner] Support for specifying tests to include with positional
arguments

* [web-platform-tests] "python serve.py" and "python manifest.py" have
now been replaced by "./serve" and "./manifest" in the root directory,
respectively. Similarly the lint is now replaced with "./lint".

Received on Tuesday, 24 February 2015 17:00:47 UTC