Re: Knowing which tests are in the repository

On Fri, Aug 23, 2013 at 3:58 AM, James Graham <james@hoppipolla.co.uk>wrote:

> On 22/08/13 23:10, Tobie Langel wrote:
>
>> On Thursday, August 22, 2013 at 11:41 PM, Dirk Pranke wrote:
>>
>>> On Thu, Aug 22, 2013 at 2:27 PM, Kevin Kershaw
>>> <K.Kershaw@cablelabs.com (mailto:K.Kershaw@cablelabs.**com<K.Kershaw@cablelabs.com>)>
>>> wrote:
>>>
>>
>
>>> However, I think you raise a valid question: if we did support
>>> manifests, would we require that all tests be in a manifest? Here
>>> there's two possible interpretations: you can't run a test at all
>>> if it's not in the manifest, or you can run the test interactively,
>>> but a test harness/runner might ignore it (and thus we might want a
>>> commit check to enforce this). Or, of course, being in the manifest
>>> could be strictly optional.
>>>
>>
>> We want to avoid test never getting run because they weren't added to
>> the manifest (or they were, but the url has a typo).
>>
>
> I feel like I didn't explain my previous proposal very well.
>
> Automation harnesses need a manifest or some other external way of storing
> which tests exist and what settings they have. That is, I think,
> uncontroversial. It is simply not viable to parse that information out of
> the test files when they are actually being run.
>
>
It's at least a little controversial :).

The way Blink and WebKit's test harness works (for their own checked-in
tests, not the W3C's), we walk a directory looking for files that have
specific file extensions and aren't in directories with particular names.
All (and only) the matches are tests; end of story. References can be found
either by filename convention or (in a few rare cases not really used
today) by parsing a reftest manifest. I think we really only support the
manifest format for feature completeness and under the belief that we would
need it sooner or later when importing the W3C's tests. It was (and
remains) a fairly controversial practice compared to using filename
conventions.

We handle the timeout problem as follows: First, we expect tests to be fast
(sub-second), and we don't expect them to timeout regularly (since running
tests that timeout don't really give you much of a signal and wastes a lot
of time).

Second, we pick a default timeout. In Blink, this is 6 seconds, which works
well for 99% (literally) of the tests on a variety of hardware (this number
could probably be a couple seconds higher or lower without much impact),
but the number is adjusted based on the build configuration (debug builds
are slower) and platform (android is slower) . Second, We have a separate
manifest-ish file for marking a subset of tests as Slow, and they get a 30s
timeout. In WebKit, we have a much longer default timeout (30s) and don't
use Slow markers at all.

There is no build step, and no parsing of tests on the fly at test run time
(except as part of the actual test execution, of course). It works well,
and any delays caused by scanning for files or dealing with timeouts is a
small (1-3%) part of the total test run.

More importantly, it's very easy to understand and very transparent.

I write all this not to argue that this is way we must do things; I
recognize that a lot of discussion has preceded me getting involved in this
group. However, I do think we have a ways to go before the tests are being
run as part of well-oiled machines by multiple browsers, and I'm trying to
provide feedback based on what we've found to work well and be liked by
developers.

The main point is, the more things we can make work by convention and
reasonable default, the better, and if we can extract more things at little
performance cost by computers, we should do so.

At any rate, I haven't yet given much thought to the rest of your email;
the examples you cite are quite helpful and I will take a look.

-- Dirk