Re: Who currently executes the tests in the w3c repos?

On 08/07/2013 03:26 PM, Dirk Pranke wrote:
> On Wed, Aug 7, 2013 at 3:06 PM, James Graham <james@hoppipolla.co.uk
> <mailto:james@hoppipolla.co.uk>> wrote:

> While I would like to hope that tests being written in Blink are
> spec-compliant and will work on multiple browsers, I have no real belief
> that that is true yet. Also, I can easily believe that the bar to
> landing tests in the w3c repos is (and should be) higher than the bar
> for landing tests in Blink.

Certainly the bar is a little different; for example you can't use 
browser-specific features when submitting the tests to web-platform-tests.

> For example, we have tests for things like Regions or Shadow DOM that
> have been developed on Blink; we might like to submit those tests to the
> W3C. I do not expect such tests to be automatically approved, given the
> state of the specs and the implementation in other browsers (we might
> want to automatically approve them, to help seed the repos and increase
> coverage, but that's different from really believing that those tests
> are good).

For these cases I think it is OK to submit and approve the tests as long 
as they match the current state of the spec. You do of course need to 
maintain the tests as the spec evolves.

> During that time, I'd like to continue running the tests to validate our
> implementation.

I don't think those things have to be contradictory.

> I can mirror those tests in the Blink repo, but that encourages Blink
> devs to develop there first, and introduces overhead in trying to keep
> the two copies in sync. Ideally I think we'd like to move to a model
> where we were authoring tests and submitting them through the W3C first.
> Perhaps I am being overly optimistic?

I hope not, because I want to move to the same model ;)

> The same could be true for tests for existing features that we would
> like to submit to the W3C. I would just as soon delete them from the
> Blink repo as soon as we've submitted them to the W3C, so that they
> don't need to coexist in both systems. And yet, it seems quite likely to
> take a while to review such tests and get them approved.

Existing tests are indeed harder. I think that if the tests aren't being 
updated that often it's fine for them to coexist in two places for a while.

>         It seems like the CSSWG's model works somewhat better if tests
>         persist
>         in the "submitted-but-not-approved" state for a significant
>         period of
>         time (which seems quite plausible for some tests, especially for
>         specs
>         that are still being worked on). Does that sound like I have
>         things right?
>
>
>     I think the assumption that it is OK for tests to be submitted but
>     not reviewed for a long time is wrong. The same standards apply here
>     as for any other code.
>
>
> At least in Chromium (but also in Mozilla, from my limited experience)
> it is not at all uncommon for some patches to be in a
> submitted-but-not-approved-and-committed state for substantial lengths
> of time (weeks or more). I would expect this to be even more true for
> tests for specs that are in flux or development, and for things that
> require buy in from multiple independent groups. In other words, I bet
> you my assumption is either correct, or we'll be approving things that
> really shouldn't be approved or have buy in.

So, my feeling is that, from the point of view of a browser vendor, you 
want the tests for a feature to land at the same time as the feature 
itself. Of course there are patches that spend a lot of time in code 
review because they are big and need substantial iteration. That's not a 
problem. What is a problem is no one doing review because everyone is 
running some heavily forked version of the repo with either totally 
unreviewed tests, or tests that they have only reviewed internally, but 
not externally. So this system should provide more motivation to 
actually deal with the review queue, without causing any more extra 
work; it's just moving the review to a different place.

Received on Wednesday, 7 August 2013 22:43:40 UTC