Re: Point Events test submission/approval process on GitHub

Thanks Matt.

Having a single repository for all tests is definitely not ideal from a
contributor's standpoint. Discoverability is not really easier than having
a separate repo, as long as all test repos are in the same account, and use
consistent naming. But even if it were, that would be dwarfed in comparison
to the notification hell that this has caused. The only possible way to get
sane notifications is for the W3C to build its own notification service
that listens for GitHub events, filters based on directories, and then
sends custom notifications via email.

The fact that issues are enabled is also confusing. Are we filing issues
for bugs in tests or tests that need to be written? If so, they'll be hard
to track. If not, having issues enabled is just an invitation for
incorrectly placed information.

As far as the review process goes, Arthur submitted a pull request [1] 5
weeks ago. It received one comment and has just sat there since. I
submitted a pull request [2] 4 weeks ago, which addressed the one comment
on Arthur's submission and added several more tests (everything that was
already in the jQuery folder in hg). There has been no activity since. How
do the Test Facilitators know when a submission has been reviewed and is
ready to be merged? Is any member from our group allowed to give their
approval or is there a specific representative that must signal approval
from the group?

Most of this is W3C process related, not specific to our group, so I'd be
happy to discuss this with someone from the appropriate team.

I've been waiting for my pull request to be merged before working with Dave
Methvin since it doesn't make sense to have conflicting pull requests open.
On a related note, we should probably just close Arthur's pull request if
we have consensus that my pull request fully covers the test assertions
from his.

[1] https://github.com/w3c/web-platform-tests/pull/104
[2] https://github.com/w3c/web-platform-tests/pull/107


On Tue, Jun 4, 2013 at 4:35 AM, Matt Brubeck <mbrubeck@mozilla.com> wrote:

> In our last call, I promised to write up a proposal for how people should
> contribute test cases to the W3C Pointer Events working group, and how we
> will approve those tests, now that we have decided to move our test suite
> from dvcs.w3c.org/hg to GitHub.  Here's the promised proposal; sorry for
> the delay!
>
> Our main goals are to encourage test submissions from all members of the
> web community by making the submission process easy and familiar, and to
> use these submissions to create a high-quality, well-reviewed test suite
> for our specifications.
>
> Our tests will be stored in a directory [1] in the w3c/web-platform-tests
> repo [2].  This helps make our test suite easy to discover and contribute
> to, since it shares a location and process with other W3C web platform test
> suites.
>
> We will adopt the submission and approval process used by the Webapps WG
> [3][4], since we don't want to make test submitters learn a different
> process for each working group.  We will document this process in the
> Pointer Events WG wiki, and link to the documentation from the Pointer
> Events specification.  The Test Management Task Force [5] is working on
> moving this process documentation to a central location [6].  We will
> provide help and feedback on that effort as needed, and change our
> documentation to point to the new centralized documentation once it's ready.
>
> Quick summary of the process:
> 1) Anyone may submit a test by creating a pull request for the GitHub repo.
> 2) The submitter may email the working group to announce or discuss the
> submission.
> 3) Any member of the working group may review the test and note any
> issues, or their approval, by commenting on it via GitHub.
> 4) A Test Facilitator from the WG will merge the pull request once the
> reviewers are satisfied that there are no remaining issues to be addressed.
>
> One practical concern I have is that it will be hard to filter out Pointer
> Events test submissions from all the other submissions to the same repo.
>  To make it easier for reviewers to be notified of submissions without
> "watching" the whole repository, I suggest that we tweak this process
> slightly by making step 2 mandatory rather than optional.  (Or, we could
> talk to the Test Management TF about whether it's better to split the web
> platform test repo into multiple repos.)
>
> If the group accepts this process, I'll start migrating our existing test
> submissions to GitHub by filing a pull request for each one (or by getting
> the originally submitters to file the pull requests). I'll also work with
> Dave Methvin to migrate the submissions from his Test The Web Forward repo
> [7].
>
> [1] https://github.com/w3c/web-**platform-tests/tree/master/**
> pointerevents<https://github.com/w3c/web-platform-tests/tree/master/pointerevents>
> [2] https://github.com/w3c/web-**platform-tests/<https://github.com/w3c/web-platform-tests/>
> [3] http://testthewebforward.org/**resources/github_test_**submission.html<http://testthewebforward.org/resources/github_test_submission.html>
> [4] http://www.w3.org/wiki/**Webapps/Testing<http://www.w3.org/wiki/Webapps/Testing>
> [5] http://www.w3.org/wiki/**Testing/Test_Management_TF<http://www.w3.org/wiki/Testing/Test_Management_TF>
> [6] http://lists.w3.org/Archives/**Public/public-test-infra/**
> 2013AprJun/0066.html<http://lists.w3.org/Archives/Public/public-test-infra/2013AprJun/0066.html>
> [7] https://github.com/dmethvin/**pointerevents-test/<https://github.com/dmethvin/pointerevents-test/>
>
>