Re: Point Events test submission/approval process on GitHub

(note: I am not on this mailing list, so please CC me on any followups).

> Having a single repository for all tests is definitely not ideal from a
> contributor's standpoint. Discoverability is not really easier than having
> a separate repo, as long as all test repos are in the same account, and use
> consistent naming. But even if it were, that would be dwarfed in comparison
> to the notification hell that this has caused. The only possible way to get
> sane notifications is for the W3C to build its own notification service
> that listens for GitHub events, filters based on directories, and then
> sends custom notifications via email.

Actually, we already have that infrastructure.

web-platform-tests is set up to use a the Opera Critic ("critic") code 
review tool, hosted at [1]. Reviews in critic are automatically assigned 
based on path filters. This means that a person who was only interested 
in pointer events tests could set up a filter for /pointer-events/ (or 
whatever the directory name is) and get a notification only for the 
changes that affect that subdirectory. This scales in both directions so 
that you can subscribe to the whole repo, or any subset, down to the 
file level.

> As far as the review process goes, Arthur submitted a pull request [1] 5
> weeks ago. It received one comment and has just sat there since. I
> submitted a pull request [2] 4 weeks ago, which addressed the one comment
> on Arthur's submission and added several more tests (everything that was
> already in the jQuery folder in hg). There has been no activity since. How
> do the Test Facilitators know when a submission has been reviewed and is
> ready to be merged? Is any member from our group allowed to give their
> approval or is there a specific representative that must signal approval
> from the group?

I expect the reason for this is that no one is actually doing the work. 
Of course we can have a super-streamlined process, but if no one is 
putting in the effort to do code review then no code will get reviewed.

If the test author/reviewers choose to use critic to do the review, 
there is an "accepted" state when all code has been reviewed and all 
issues have been addressed, so it is super-simple to know when 
everything is ready to merge. If they decide to use the github UI 
directly, it needs someone to say the equivalent of "looks good to me".

> On Tue, Jun 4, 2013 at 4:35 AM, Matt Brubeck <mbrubeck@mozilla.com> wrote:

>> Our tests will be stored in a directory [1] in the w3c/web-platform-tests
>> repo [2].  This helps make our test suite easy to discover and contribute
>> to, since it shares a location and process with other W3C web platform test
>> suites.

That sounds like an excellent plan to me. The goal is to have everyone 
share as much as possible rather than have as many different processes 
as there are specifications. As you rightly point out this dramatically 
reduces the friction for people looking to contribute tests, and, 
equally importantly, makes it much easier for implementors looking to 
actually run the tests as part of their automated testing.

[1] https://critic.hoppipolla.co.uk (uses github/OAuth for authentication)

Received on Tuesday, 4 June 2013 13:04:53 UTC