Re: Review of tests upstreamed by implementors

On 3/20/13 2:05 AM, "Tobie Langel" <tobie.langel@gmail.com> wrote:

>Hi, 
>
>As pointed out by Robin[1], the test review process is more often than
>not the bottleneck.
>
>I've had a number of offline conversations about lightening the review
>process for tests as much as possible.
>
>One of the topic that came up on multiple occasions was that tests
>upstreamed by implementers had already been peer-reviewed internally.
>
>It seems giving special treatment to such submissions would help reduce
>the bottleneck and get tests in the repository much faster. (Note that we
>could still run a number of tests automatically on such submission to
>catch common issues).
>
>In order to go through this fast-track process, some form of log of the
>internal review process would need to be produced alongside the
>submission. For open-source projects, this could be an URL to a publicly
>accessible bug tracker, for non open-source projects, this would need to
>be added to the body of the pull request.
>
>Should a given submission prove problematic, the merge would be reverted
>and the tests would go through the regular review process.

Yes!!  Thank you for bringing this up, Tobie. I've been thinking about
this lately wrt the CSS test suites as well.  This hits on another issue
that has also been a bit of a bottleneck which is the requirement* that a
review is done by someone outside of your company/organization. If we
agree to implement the fast-track process that you laid out, it seems
we'll have to make a decision or statement about same-company reviews as
well. If Company A writes a bunch of tests that Company A is implementing,
it's highly likely that someone from Company A is also doing the reviews
(the only situations where this isn't in fact guaranteed are open source
projects, but still very likely there).  And for either of these potential
changes in process, are these agreed upon at the WG or W3C level?

* I've been involved with writing W3C tests for a little over a year and
I've understood this to be a "requirement" purely from word of mouth. I
haven't seen it written anywhere (I've looked), but since it's often tough
to find info, it doesn't mean it isn't. If anyone has a pointer to where
this is documented, please share.  Anyway, my colleagues and I have been
working under this rule all this while and thus have hundreds of tests
sitting in a queue for many months. They're not (yet) submitted to Webkit,
but if/when they are, they'll likely be reviewed there by someone who
isn't allowed to be the reviewer in the W3C because he sits the next cube
over from the test author.  Would the Webkit review then be acceptable?  I
was also never clear if this rumored requirement was just for my WG or for
others. Folks from other WGs - let me know if if this isn't a constraint
in your review process. If it turns out to be just a thing in the CSSWG,
I'll raise this issue there and point to this thread.

-Rebecca

Received on Wednesday, 20 March 2013 16:51:53 UTC