Re: Knowing which tests are in the repository

On Friday, August 23, 2013 at 12:58 PM, James Graham wrote:
> I feel like I didn't explain my previous proposal very well.
> 
> Automation harnesses need a manifest or some other external way of 
> storing which tests exist and what settings they have. That is, I think, 
> uncontroversial. It is simply not viable to parse that information out 
> of the test files when they are actually being run.

Agreed, but this feels somewhat of an implementation detail of the test runner. i.e. I'm not sure what the benefit is of standardizing this. 
> The only remaining question is when that manifest is created. It can 
> either be created by test authors explicitly, or created by people 
> running the tests when they update their copy of the repository. I am 
> proposing the latter.

Good. And I'm strongly in favor of the latter solution too. This feels like something a computer would do a lot more reliably than a human being. 
> However there are special cases where I maintain 
> that an explicit manifest is useful to solve real problems in a 
> straightforward way. These are not typical cases, but they do occur.

Understood. 

If you accept the premise that the format of the manifest containing all the necessary metadata to run the test properly can be an implementation details of the test runner, it follows that the explicit manifest you mention above could be either specific to solving the real problems you brought up (timeout, query strings in url) or generic and used as both an input for these explicit needs AND the output of the automated manifest generating process (with the risk of having duplicated and out of sync content in it, e.g. type: reftest with a reference to testharness.js).

I would prefer the first option: keep the metadata collection an implementation detail of the test runner, and allow a very specific manifest format to handle the small set of fringe use cases you brought up.

--tobie

Received on Friday, 23 August 2013 12:43:28 UTC