Re: Proposal to introduce test suite curators

On 03/03/2017 04:36 PM, Gérard Talbot wrote:
> Le 2017-03-02 21:06, Florian Rivoal a écrit :
>> We suffer from a lack of review on tests. The average age of a PR on
>> the test repo is currently around 370 days, or 195 days even if we
>> only count those not marked "awaiting-submitter-response". That's way
>> too long.
>>
>> I think that's in part because nobody in particular feels responsible for this.
>
> Reviewing tests that others have written and submitted
> a) requires time, efforts, care; you have to report your findings in an email and/or in Shepherd system, then propose
> solutions, constructive modifications with tact, diplomacy
> b) is not necessarly fun by itself, of itself: reviewing a lot of tests has also been a learning experience as I was able to
> see how others (more experienced test authors) build, design tests
> c) requires skills, experience of doing so eventually, and
> d) requires good understanding of the involved specification(s) to begin with. There are tests that have been submitted and
> reviewed and which are incorrect; overall, rare but it happened.
>
>> I do not think we can force anyone to review tests if they'd rather be
>> doing something else, but we could work on identifying which spec lack
>> test reviewers, and try to find volunteers.
>
> Volunteers: if the review process is about finding, reporting
> a) incorrect tests,
> b) tests that can not fail
> c) tests that do not test what they claim (or believe) to be testing
> d) imprecise tests, ill-designed tests and
> e) how to improve, rehabilitate a)-to-d) types of tests,
> then I think you can *not* expect volunteers to do this.

I feel it's necessary to point out here that some of the top Web standards
QA people from the early 2000s era were volunteers who got involved through
Mozilla and Opera's volunteer standards-compliance QA communities.  They
eventually got hired, but not before establishing a solid track record of
high-caliber work... which is to say that volunteers are absolutely capable
of doing such work, if they are so inclined.

~fantasai

Received on Monday, 13 March 2017 22:39:16 UTC