Re: Proposal to introduce test suite curators

I think it would be good if the curators were not just the editors (to avoid guessing what is in the spec instead of reading it), and it would be good to have implementers involved there, for the reason you say.

However, since we're generally facing a shortage of reviewers, I would say it should just be a preference when the chairs try and recruit curators rather than an absolute rule.

—Florian
> On Mar 3, 2017, at 15:33, Koji Ishii <kojiishi@gmail.com> wrote:
> 
> What about requiring one implementer, instead of or in addition to, whichever we prefer?
> 
> When implementers are working on a feature, they are very motivated to review, and also they are very good reviewers since they look things from different perspective.
> 
> I saw several cases where tests written before even the first impl are often not correct, even after reviewer's approval.
> 
> Years later, when someone try to implement, find questions to tests, or tests don't look correct, but test authors tend to forget what it was, do not respond, or mail returned in error.
> 
> So I think requiring an implementer can improve both review speed and tests quality. WDYT?
> 
> 2017-03-03 11:06 GMT+09:00 Florian Rivoal <florian@rivoal.net <mailto:florian@rivoal.net>>:
> We suffer from a lack of review on tests. The average age of a PR on the test repo is currently around 370 days, or 195 days even if we only count those not marked "awaiting-submitter-response". That's way too long.
> 
> I think that's in part because nobody in particular feels responsible for this.
> 
> I do not think we can force anyone to review tests if they'd rather be doing something else, but we could work on identifying which spec lack test reviewers, and try to find volunteers.
> 
> I think each spec needs a minimum of 2 test reviewers. They can be the same as the editors, but do not have to be. I think it has to be at least 2, because if there's only one, there's nobody to review that person's tests, even though they are fairly likely to write some.
> 
> So I suggest we introduce a new role, which I'll call Test Curator (feel free to bikeshed). The primary responsibility is not to write the entire test suite, but to ensure that reviews and merges of Pull Requests get done in a timely manner. Secondary responsibilities could include: being able to identify which are of the test suite need more work (or to declare it complete) or checking if existing tests are still valid after normative changes in the spec.
> 
> I suggest that on each spec:
> 
> * we should list the "Test Curators" , next to and separately from the Editors.
> 
> * Ask Editors if they're willing to be a Test Curator as well. If they are, list them as such.
> 
> * For Every spec that has less than 2 test curators, call for volunteers
> 
> * Have the chairs periodically (monthly?) check if more people need to be appointed (in addition or instead) for specs which still have a large queue of Pull Requests. This should be take as seriously as finding new Editors for a spec when the previous/current ones have left or aren't keeping up.
> 
> Note that this is separate from the (poorly named) OWNERS file used in the web-platform-test repo, which is merely a notification mechanism, with no implied responsibility.
> 
> —Florian
> 

Received on Friday, 3 March 2017 07:52:36 UTC