Sharing reference files (was: Knowing which tests are in the repository)

On 8/22/13 8:18 AM, "Robin Berjon" <robin@w3.org> wrote:

>On 21/08/2013 19:10 , James Graham wrote:
>> FWIW I just tested this (in a not-very-careful way) on the
>> web-platform-tests repository and it seems that, even though there are
>> very few reftests at the moment we could have saved 17 screenshots from
>> the 338 tests without any extra effort. This number would undoubtedly
>> increase if an effort was made to consolidate the references to be the
>> same for more tests.
>
>Reducing the number of screenshots is an interesting optimisation, but
>one that can be equally well achieved with a quick hash of the source or
>some similar technique. I don't think we should drive our choices off
>this sort of optimisation.

Aside from the optimization, reusing ref files has other common-sense
benefits. Less files is good. Having a suite of related tests designed
around the same visual concept (rather than designing each individually)-
that's good, too. Particularly if using a common green-square-no-red or
similar reference, new tests are steered toward our recommended good
green/red style. All of these things make reviewing a little easier.

However, I've personally been situations where I've found myself straining
a bit too hard to make all of my tests fit into the same (green) box.
Sometimes it can be challenging to do while keeping your tests clean and
minimal. If I find myself having to add a whole bunch of CSS (or whatever)
that isn't directly related to what I'm testing yet only necessary to
match the ref, I back out and create a new ref. Same if it's something
that takes more than 15 minutes figure out how to set up. I think these
are more important things to consider when deciding when to reuse or not.

I debated writing about this in the docs and decided not to overcomplicate
the matter by leaving it as one suggestive ("preferable") sentence rather
than a mandate. If anyone thinks extra guidance on when to reuse refs
should be added to the docs, speak up. I'm kind of on the fence about it.

Received on Thursday, 22 August 2013 22:08:38 UTC