Re: Knowing which tests are in the repository

On Fri, Aug 23, 2013 at 6:12 AM, James Graham <james@hoppipolla.co.uk>wrote:

> On 23/08/13 13:43, Tobie Langel wrote:
>
>> On Friday, August 23, 2013 at 12:58 PM, James Graham wrote:
>>
>>> I feel like I didn't explain my previous proposal very well.
>>>
>>> Automation harnesses need a manifest or some other external way of
>>> storing which tests exist and what settings they have. That is, I
>>> think, uncontroversial. It is simply not viable to parse that
>>> information out of the test files when they are actually being
>>> run.
>>>
>>
>> Agreed, but this feels somewhat of an implementation detail of the
>> test runner. i.e. I'm not sure what the benefit is of standardizing
>> this.
>>
>
> The benefit is that it's clear everyone agrees on which tests are in the
> repo and what properties they have. If we have N different implementations
> that could lead to N different conclusions about what tests we have and
> what their properties are.
>
> Of course people are free to use their own implementation, as long as we
> have some way of agreeing what the correct output is. Therefore I will
> implement the proposal, check in the result, and let people either use it
> directly, or check their own tools against it.
>

I'm inclined to agree that if we thought that every test harness needed a
manifest of some form, we should probably standardize on the format.
 Obviously, I'm not yet convinced that we do need a manifest as part of the
files in the repo, though :).

-- Dirk