Re: Knowing which tests are in the repository

On 23/08/13 13:43, Tobie Langel wrote:
> On Friday, August 23, 2013 at 12:58 PM, James Graham wrote:
>> I feel like I didn't explain my previous proposal very well.
>>
>> Automation harnesses need a manifest or some other external way of
>> storing which tests exist and what settings they have. That is, I
>> think, uncontroversial. It is simply not viable to parse that
>> information out of the test files when they are actually being
>> run.
>
> Agreed, but this feels somewhat of an implementation detail of the
> test runner. i.e. I'm not sure what the benefit is of standardizing
> this.

The benefit is that it's clear everyone agrees on which tests are in the 
repo and what properties they have. If we have N different 
implementations that could lead to N different conclusions about what 
tests we have and what their properties are.

Of course people are free to use their own implementation, as long as we 
have some way of agreeing what the correct output is. Therefore I will 
implement the proposal, check in the result, and let people either use 
it directly, or check their own tools against it.

Received on Friday, 23 August 2013 13:15:35 UTC