Re: Knowing which tests are in the repository

On 15/08/13 20:42, Tobie Langel wrote:
> On Thursday, August 15, 2013 at 5:27 PM, James Graham wrote:
>> As far as I know we still haven't solved the problem of determining
>> which files in the repository are tests and of what type.
>
> Yup.
>> As I
>> understand it there have been two serious proposals:
>>
>> a) Use file naming conventions
>> b) Use a manifest
>>
>> I think b) has a number of critical advantages, which outweigh the
>> undoubted ease of authoring of a). Notably we can store extra metadata
>> which is needed to run the tests such as the comparison type for
>> reftests
>
> That's already included within the reftests themselves.
>> and unusually long timeouts for testharness tests.
>
> Doesn't testharness already let you handle timeouts?

Sure. But the general problem is this:

I am writing a test runner for these tests. In order for this to work I 
need to know at least which files are tests. But for a good 
implementation I also need various other pieces of data about the tests. 
For example I need to know the timeouts because I want the runner to 
have it's own timeout in case the test causes the browser to hang or 
crash. This obviously needs to be at least as long as the test timeout. 
Therefore the timeout in the test is relevant, but the fact that 
testharness.js will timeout itself isn't really relevant.

Now, it is true that for some of the information I could parse it out of 
the files. But if I do that I am certainly going to store it in some 
intermediate format so that I don't have to redo the expensive parsing 
step for each test run. So at some level I am going to end up with a 
manifest format. Since everyone else trying to run the tests in bulk 
will also need this it makes sense to share the format and the files.

Moreover, the filename-only solution doesn't work. You don't load files, 
you load URLs. A particular file can have different behavior depending 
on non-path parts of the URL. For example the parser tests work 
differently depending on the query string that you pass in.

Received on Thursday, 15 August 2013 20:27:12 UTC