Plan for reftests

In order to better align with what CSS do and make reftests more
flexible, I plan to implement some changes to the way that reftests work
in the future:

* Reftests will be identified as files that are not other kinds of tests
and contain <link rel=match> or <link rel=mismatch>.

* Multiple such <link> elements in a file will have OR semantics i.e. in
order for the test to pass it much match one of the two conditions. This
is expected to be rare but is sometimes needed where specs leave things
underdefined.

* In order to get AND semantics, refs may themselves <link> to other refs.

* In order to distinguish a file that should be considered a test from
one that should be considered a pure reference, any reference containing
<link rel=[mis]match> elements must have -ref or -notref at the end of
the filename, before the extension.

* Reference cycles will be permitted e.g. A == B == C == B. It is not
quite clear what to do about cycles including the test itself; in the ==
case these seem likely to be bugs e.g A == A or A == B == A. However
there are != cases where this makes sense e.g. A == B != C != A.

* When executing, the graph formed with the resources as vertexes and
edges as <link>s will be flattened into a tree by breaking cycles. This
tree will be processed in a depth-first search with the goal of making
it to a leaf vertex along only edges where the comparison succeeds. If
such a leaf is found the comparison terminates early. If Something Bad
happens before such an edge is reached (e.g. a browser crash or an error
of some kind) the test will terminate with that status.

Does that make sense?

Received on Monday, 26 January 2015 17:19:57 UTC