Re: Knowing which tests are in the repository

On 16/08/13 20:33, Dirk Pranke wrote:

> On Fri, Aug 16, 2013 at 2:51 AM, James Graham <james@hoppipolla.co.uk
>         In general, I think it's much easier to understand the layout of the
>         repository if we follow naming conventions, rather than requiring
>         lookups through a manifest. To that end, Blink and WebKit have
>         always
>         preferred the -expected or -ref convention for references over the
>         "rel=match" metadata approach, but I think that ship has sailed
>         anyway.
>
>
>     That was always a bad idea. One nice thing about reftests is that
>     you can reuse a single ref for many tests. This is not only less
>     work for the tester, but means that you can optimise your test
>     runner to only take one screenshot per refurl. Requiring strict
>     naming conventions defeats this.
>
>
> I don't think the tradeoffs are quite that clear. There are lots of
> advantages to having the reference be obvious in the file system, and
> it's not clear to me how many references are reused such that this is
> really a big performance win. But, as I said, I'm not trying to re-open
> this particular debate.

FWIW I just tested this (in a not-very-careful way) on the 
web-platform-tests repository and it seems that, even though there are 
very few reftests at the moment we could have saved 17 screenshots from 
the 338 tests without any extra effort. This number would undoubtedly 
increase if an effort was made to consolidate the references to be the 
same for more tests.

For the CSS tests I would expect the effect to be much larger, 
particularly if there is an effort to reuse the same few patterns for 
tests. There are a lot of things that can be tested with "There should 
be a green square below" or "It should say 'PASS' below". I seem to 
recall that when Opera converted pure visual tests to reftests we made 
an effort to standardise on a few references and saved a huge chunk of 
test time in so doing.

If we are not already, we should encourage reftest authors to reuse 
references wherever possible rather than creating new files.

Received on Wednesday, 21 August 2013 17:11:02 UTC