Re: Code review improvements

On 11/21/13 4:42 AM, ext Dominique Hazael-Massieux wrote:
> On jeu., 2013-11-14 at 11:17 +0700, James Graham wrote:
>> Given that there is a history of vendors submitting tests that do in
>> fact match their implementations but do not in fact match the spec, and
>> tests that have significant style issues, it does seem that some
>> mechanism is needed to allow external review of these tests. Two
>> possibilities have been suggested so far; that the tests only get
>> approved after some timeout (i.e. review can be carried forward, but we
>> wait for N days before actually merging to give a chance for people to
>> review), or that we keep an out-of-band record of which PRs were only
>> reviewed within an organization and allow people to do explicit post-hoc
>> review. I don't know which of these options is better; or if there are
>> further options. I think the second has some advantages for integrating
>> with UA import systems since it keeps the code more in-sync between the
>> UA repo and the external repo. Both also require some additional
>> tooling.
>>
>> Comments?
> I too slightly prefer the second option for the same reason. A couple of
> additional thoughts:
> * I think WGs have an incentive to check that the submitted tests match
> the specs, esp. to go out of CR; we should use that incentive to
> increase their involvement in test reviews of externally submitted tests

I agree with your sentiment but (unfortunately) in practice, I don't see 
"getting to REC" as a sufficient incentive for most participants.

I don't know how to change that dynamic. The argument that IP 
commitments by WG members are not finalized until the REC is published 
is of course true but it practice, I can't think of any cases (in the 
context of OWP specs) where the lack of a REC actually prevented someone 
from implementing or deploying a spec.

There are of course other reasons RECs are useful for some stakeholders 
(f.ex. to meet procurement or legislative requirements) but again, those 
reasons don't appear to be generally sufficient to get people to review 
tests.

> (e.g. this could be something that the Director explicitly asks during
> transition out of CR)

Indeed "Who reviewed the tests?" is a fair question to ask.

-AB

Received on Thursday, 21 November 2013 15:01:16 UTC