Re: Who currently executes the tests in the w3c repos?

On Wed, Aug 7, 2013 at 3:06 PM, James Graham <james@hoppipolla.co.uk> wrote:

> On 08/07/2013 02:19 PM, Dirk Pranke wrote:
>
>>
>> I wasn't aware of that, thanks.
>>
>> That said, having taken a quick look, that seems like that is quite a
>> bit less accessible, being N different copies of the main repo (1 per
>> pull request / patch), rather than having the specifically submitted
>> files all available alongside the approved ones.
>>
>
> Think of it as encouragement to do test review :)
>
> I presume you typically review Google-written (e.g. gtest) tests before
> they are committed to your repository. This is important for lots of
> reasons, for example ank the void bad/flaky tests breaking the build.

 You can apply the same process to web-platform-tests tests, but with the
> proviso that the review has to be done either on github or in critic. It
> can be done by the blink contributer that would do the review anyway, just
> in a public location.


While I would like to hope that tests being written in Blink are
spec-compliant and will work on multiple browsers, I have no real belief
that that is true yet. Also, I can easily believe that the bar to landing
tests in the w3c repos is (and should be) higher than the bar for landing
tests in Blink.

For example, we have tests for things like Regions or Shadow DOM that have
been developed on Blink; we might like to submit those tests to the W3C. I
do not expect such tests to be automatically approved, given the state of
the specs and the implementation in other browsers (we might want to
automatically approve them, to help seed the repos and increase coverage,
but that's different from really believing that those tests are good).

During that time, I'd like to continue running the tests to validate our
implementation.

I can mirror those tests in the Blink repo, but that encourages Blink devs
to develop there first, and introduces overhead in trying to keep the two
copies in sync. Ideally I think we'd like to move to a model where we were
authoring tests and submitting them through the W3C first. Perhaps I am
being overly optimistic?

The same could be true for tests for existing features that we would like
to submit to the W3C. I would just as soon delete them from the Blink repo
as soon as we've submitted them to the W3C, so that they don't need to
coexist in both systems. And yet, it seems quite likely to take a while to
review such tests and get them approved.

It seems like the CSSWG's model works somewhat better if tests persist
>> in the "submitted-but-not-approved" state for a significant period of
>> time (which seems quite plausible for some tests, especially for specs
>> that are still being worked on). Does that sound like I have things right?
>>
>
> I think the assumption that it is OK for tests to be submitted but not
> reviewed for a long time is wrong. The same standards apply here as for any
> other code.
>

At least in Chromium (but also in Mozilla, from my limited experience) it
is not at all uncommon for some patches to be in a
submitted-but-not-approved-and-committed state for substantial lengths of
time (weeks or more). I would expect this to be even more true for tests
for specs that are in flux or development, and for things that require buy
in from multiple independent groups. In other words, I bet you my
assumption is either correct, or we'll be approving things that really
shouldn't be approved or have buy in.

That said, we can certainly try things as they are and see how it goes.

-- Dirk