Re: Code review improvements

On jeu., 2013-11-14 at 11:17 +0700, James Graham wrote:
> Given that there is a history of vendors submitting tests that do in 
> fact match their implementations but do not in fact match the spec, and 
> tests that have significant style issues, it does seem that some 
> mechanism is needed to allow external review of these tests. Two 
> possibilities have been suggested so far; that the tests only get 
> approved after some timeout (i.e. review can be carried forward, but we 
> wait for N days before actually merging to give a chance for people to 
> review), or that we keep an out-of-band record of which PRs were only 
> reviewed within an organization and allow people to do explicit post-hoc 
> review. I don't know which of these options is better; or if there are 
> further options. I think the second has some advantages for integrating 
> with UA import systems since it keeps the code more in-sync between the 
> UA repo and the external repo. Both also require some additional 
> tooling.
> 
> Comments?

I too slightly prefer the second option for the same reason. A couple of
additional thoughts:
* I think WGs have an incentive to check that the submitted tests match
the specs, esp. to go out of CR; we should use that incentive to
increase their involvement in test reviews of externally submitted tests
(e.g. this could be something that the Director explicitly asks during
transition out of CR)

* I wonder how much of the stylistics checks could be automated or
partially automated; is there anywhere one could find a list of frequent
such errors that have been raised in previous test reviews?

Dom

Received on Thursday, 21 November 2013 09:42:39 UTC