Re: Knowing which tests are in the repository

On Thursday, August 15, 2013 at 10:26 PM, James Graham wrote:

> On 15/08/13 20:42, Tobie Langel wrote:
> > On Thursday, August 15, 2013 at 5:27 PM, James Graham wrote:
> > > and unusually long timeouts for testharness tests.
> > 
> > Doesn't testharness already let you handle timeouts?
> 
> Sure. But the general problem is this:
> 
> I am writing a test runner for these tests. In order for this to work I 
> need to know at least which files are tests. But for a good 
> implementation I also need various other pieces of data about the tests. 
> For example I need to know the timeouts because I want the runner to 
> have it's own timeout in case the test causes the browser to hang or 
> crash. This obviously needs to be at least as long as the test timeout. 
> Therefore the timeout in the test is relevant, but the fact that 
> testharness.js will timeout itself isn't really relevant.

Can't you parse the test to get that data out?

Else couldn't we set a convention to add this info in a meta tag?
> Now, it is true that for some of the information I could parse it out of 
> the files. But if I do that I am certainly going to store it in some 
> intermediate format so that I don't have to redo the expensive parsing 
> step for each test run.

Is it really going to be that expensive? 
> So at some level I am going to end up with a 
> manifest format. Since everyone else trying to run the tests in bulk 
> will also need this it makes sense to share the format and the files.

Possibly. 
> Moreover, the filename-only solution doesn't work. You don't load files, 
> you load URLs. A particular file can have different behavior depending 
> on non-path parts of the URL. For example the parser tests work 
> differently depending on the query string that you pass in.

Pointers to these tests? This seems a rather isolated case for which there might be a dedicated solution that would work as well.

--tobie

Received on Thursday, 15 August 2013 22:30:30 UTC