Re: Knowing which tests are in the repository

On 22/08/13 23:10, Tobie Langel wrote:
> On Thursday, August 22, 2013 at 11:41 PM, Dirk Pranke wrote:
>> On Thu, Aug 22, 2013 at 2:27 PM, Kevin Kershaw
>> <K.Kershaw@cablelabs.com (mailto:K.Kershaw@cablelabs.com)> wrote:

>>
>> However, I think you raise a valid question: if we did support
>> manifests, would we require that all tests be in a manifest? Here
>> there's two possible interpretations: you can't run a test at all
>> if it's not in the manifest, or you can run the test interactively,
>> but a test harness/runner might ignore it (and thus we might want a
>> commit check to enforce this). Or, of course, being in the manifest
>> could be strictly optional.
>
> We want to avoid test never getting run because they weren't added to
> the manifest (or they were, but the url has a typo).

I feel like I didn't explain my previous proposal very well.

Automation harnesses need a manifest or some other external way of 
storing which tests exist and what settings they have. That is, I think, 
uncontroversial. It is simply not viable to parse that information out 
of the test files when they are actually being run.

The only remaining question is when that manifest is created. It can 
either be created by test authors explicitly, or created by people 
running the tests when they update their copy of the repository. I am 
proposing the latter. However there are special cases where I maintain 
that an explicit manifest is useful to solve real problems in a 
straightforward way. These are not typical cases, but they do occur.

> If we end up wanting to avoid running specific tests, I suggest tying
> that to the versioning system or the issue tracker (e.g. provide an
> option to not run files that have an issue opened against them).

That wouldn't solve any of the issues that have been brought up in this 
thread, so I'm not sure why you are proposing it. I am also not willing 
to tie our infrastructure to github more than is absolutely necessary 
(i.e. depending on being able to fetch from the repository is 
unavoidable, depending on the issue tracker is not acceptable).

Received on Friday, 23 August 2013 10:59:27 UTC