Re: "priority" of tests

On 10/05/17 20:44, Philip Jägenstedt wrote:
> I think that the "interop risk" view might be even more useful if done at
> the granularity of individual tests instead of files, although that's not
> entirely trivial because there's no guarantee that the number and names of
> tests match up. Any ideas for other ways to interrogate the data would be
> appreciated.

I think that doing less that this doesn't make much sense. A missing 
result should be treated as another status, and handled like a FAIL. I'm 
sure that would be wrong occasionally but it would be right most of the 
time.

> For some kind of metadata, would that be at the test level? At least I tend
> to write one file to test (for example) everything about a constructor, and
> that would mix the serious with the trivial in the same file. But we have
> no mechanism for annotating the individual tests.

So that's technically untrue. But nevertheless I don't think a metadata 
based system will work. Historically we have never managed to get 
developers as a group to add metadata at all — even getting something as 
basic as useful commit messages is hard — and even where individuals 
have been motivated to add it it has always bitrotted rather quickly.

I believe a plan based around getting people to add vague value 
judgements about the importance of tests would be doomed to failure. 
Even if we could get people to add this data at all, it would mostly be 
wrong when added and then later be even more wrong (because an 
"unimportant test" can be "important" when it turns out that specific 
condition is triggered in Facebook).

I wish this wasn't true, but I think the reality is that there just 
isn't a simple solution to figuring out which tests are important. Often 
it's possible to tell that some class of failures isn't urgent (because 
e.g. as Philip says you recognise that a specific failure message 
relates to a known error in your WebIDL implementation), but otherwise 
you need someone with expertise to make a judgement call.

Received on Wednesday, 10 May 2017 20:50:46 UTC