Re: Knowing which tests are in the repository

On Thu, Aug 22, 2013 at 8:18 AM, Robin Berjon <robin@w3.org> wrote:

> On 21/08/2013 19:10 , James Graham wrote:
>
>> FWIW I just tested this (in a not-very-careful way) on the
>> web-platform-tests repository and it seems that, even though there are
>> very few reftests at the moment we could have saved 17 screenshots from
>> the 338 tests without any extra effort. This number would undoubtedly
>> increase if an effort was made to consolidate the references to be the
>> same for more tests.
>>
>
> Reducing the number of screenshots is an interesting optimisation, but one
> that can be equally well achieved with a quick hash of the source or some
> similar technique. I don't think we should drive our choices off this sort
> of optimisation.
>
>
I assume by "the source" you mean the "source of the reference", but even
that that would only work for files that were exact clones of each other,
and not references that happened to differ but still produce the same
output.

[ I have a vague memory of Blink's pixel tests containing roughly ~15%
duplicates (out of about 7000 pixel tests out of ~30k tests), and a large
chunk of those being the result of old imports of CSS1 and CSS2.1 tests. It
would be fairly easy to run blink over the current CSSWG repo and see, but
I haven't had the time to do that lately. ]

Regardless, it seems unlikely to me that we'd get more than 15-20% reuse of
references, meaning that we could save at most 10% of execution time; not
something to ignore, but not something world-changing, either.

I think the optimization is a minor point, though, and we haven't really
come to an answer for James' original question.

So, to restate (in my words) the questions he posed:

1) Do we want to require (or at least strongly recommend) file naming
conventions for tests and references?

2) Do we want to require the use of a manifest as a source-level object in
the repo (rather than something that could be generated via a build
process)?

3) Do we want to allow tests that are specified only in a manifest (e.g.,
tests with query parameters) rather than being initiated from a
non-manifest file?

Obviously, if the answer to (2) is yes, then (1) becomes less important and
(3) becomes possible, but if the answer to (2) is no, then (3) is off the
table (I think, unless I'm missing something) and (1) is less compelling.

Blink and WebKit have long traditions of using auxiliary files to describe
the expected results of a test run and various pieces of metadata like
whether a test is slow or not (in Blink and recent WebKit builds, the files
are called TestExpectations; older WebKit versions used a simpler concept
called Skipped lists).

In general, having entries in those files is painful and we like to avoid
them; it causes a maintenance burden (merges, etc.) and it means you have
to look in multiple places to figure out what the tests are and how they're
expected to run. I think having tests that *only* existed in the manifest
would be particularly jarring.

[ For the record, Blink has something somewhat similar called "virtual test
suites" that are instructions to re-run tests in a particular directory
with different command line arguments to the test binary. We use this to
force different paths through the code to test different runtime
configurations like software vs. hardware compositing, etc., but this is
obviously a non-portable construct ].

For the reasons above, I don't like the idea of manifests being required as
a source construct. I'm not completely against them, but I haven't found
the reasons stated thus far to convince me that the benefits outweigh the
disadvantages. I think most Blink and WebKit developers would agree with
me. I particularly don't like the idea of having tests exist only in the
manifest file. [ That said, I still haven't really had the time to study
the URL examples James gave, I apologize; it's possible that might change
my mind, though Aryeh's response didn't help :) ].

I think having naming conventions is a good idea regardless and we should
require them as much as we can (even if we did adopt manifests).

-- Dirk