Re: Point Events test submission/approval process on GitHub

On 6/4/13 8:10 AM, ext Scott González wrote:
> Thanks Matt.

Yes, thanks Matt.

> Having a single repository for all tests is definitely not ideal from 
> a contributor's standpoint. Discoverability is not really easier than 
> having a separate repo, as long as all test repos are in the same 
> account, and use consistent naming.

I can see advantages to both creating our own repo and reusing 
w3c/web-platform-tests/. My preference is to continue with our decision 
to reuse w-p-t and only if we agree that it is unworkable do we revisit 
that decision.

To date, I think most of the PE submissions from non WG members are the 
result of a TTWF event and I believe the instructions for those event 
participants are pretty clear about the repo organization 
<http://testthewebforward.org/resources/github_test_submission.html>.


> But even if it were, that would be dwarfed in comparison to the 
> notification hell that this has caused. The only possible way to get 
> sane notifications is for the W3C to build its own notification 
> service that listens for GitHub events, filters based on directories, 
> and then sends custom notifications via email.

Yeah, I feel this pain too (because I am watching WebApps' specs). Let's 
see if Tobie and the Testing TF can help address problem. Note he just 
created <http://www.w3.org/wiki/Testing/Infra/Notification_Hell>. Please 
send comments to public-test-infra @ w3org.

> The fact that issues are enabled is also confusing. Are we filing 
> issues for bugs in tests or tests that need to be written? If so, 
> they'll be hard to track. If not, having issues enabled is just an 
> invitation for incorrectly placed information.

This seems like something we should be able to address.

> As far as the review process goes, Arthur submitted a pull request [1] 
> 5 weeks ago. It received one comment and has just sat there since. I 
> submitted a pull request [2] 4 weeks ago, which addressed the one 
> comment on Arthur's submission and added several more tests 
> (everything that was already in the jQuery folder in hg). There has 
> been no activity since. How do the Test Facilitators know when a 
> submission has been reviewed and is ready to be merged? Is any member 
> from our group allowed to give their approval or is there a specific 
> representative that must signal approval from the group?

It seems like we just need to come to some common understanding on how 
we want this to work for us. Since, ATM, we have only one spec, I'm fine 
with having multiple Test Facilitators [I think Scott's e-mail 
automatically makes him an additional Test Facilitator ;-)] and using 
our mail list for all testing related announcements (e.g. new submission 
made, request for review, etc.).

> Most of this is W3C process related, not specific to our group, so I'd 
> be happy to discuss this with someone from the appropriate team.

Yes, I think we want to use/reuse as much of the common processes as 
possible and the common issues should be discussed on public-test-infra 
list. At the same time, if we do indeed have some specific requirements 
for our work, I think it's OK to define them ourselves.


> I've been waiting for my pull request to be merged before working with 
> Dave Methvin since it doesn't make sense to have conflicting pull 
> requests open. On a related note, we should probably just close 
> Arthur's pull request if we have consensus that my pull request fully 
> covers the test assertions from his.
>
> [1] https://github.com/w3c/web-platform-tests/pull/104
> [2] https://github.com/w3c/web-platform-tests/pull/107

Yeah, that is certainly OK with me [and if I need to do something to 
help make that happen, please let me know.]

-AB

>
>
> On Tue, Jun 4, 2013 at 4:35 AM, Matt Brubeck <mbrubeck@mozilla.com 
> <mailto:mbrubeck@mozilla.com>> wrote:
>
>     In our last call, I promised to write up a proposal for how people
>     should contribute test cases to the W3C Pointer Events working
>     group, and how we will approve those tests, now that we have
>     decided to move our test suite from dvcs.w3c.org/hg
>     <http://dvcs.w3c.org/hg> to GitHub.  Here's the promised proposal;
>     sorry for the delay!
>
>     Our main goals are to encourage test submissions from all members
>     of the web community by making the submission process easy and
>     familiar, and to use these submissions to create a high-quality,
>     well-reviewed test suite for our specifications.
>
>     Our tests will be stored in a directory [1] in the
>     w3c/web-platform-tests repo [2].  This helps make our test suite
>     easy to discover and contribute to, since it shares a location and
>     process with other W3C web platform test suites.
>
>     We will adopt the submission and approval process used by the
>     Webapps WG [3][4], since we don't want to make test submitters
>     learn a different process for each working group.  We will
>     document this process in the Pointer Events WG wiki, and link to
>     the documentation from the Pointer Events specification.  The Test
>     Management Task Force [5] is working on moving this process
>     documentation to a central location [6].  We will provide help and
>     feedback on that effort as needed, and change our documentation to
>     point to the new centralized documentation once it's ready.
>
>     Quick summary of the process:
>     1) Anyone may submit a test by creating a pull request for the
>     GitHub repo.
>     2) The submitter may email the working group to announce or
>     discuss the submission.
>     3) Any member of the working group may review the test and note
>     any issues, or their approval, by commenting on it via GitHub.
>     4) A Test Facilitator from the WG will merge the pull request once
>     the reviewers are satisfied that there are no remaining issues to
>     be addressed.
>
>     One practical concern I have is that it will be hard to filter out
>     Pointer Events test submissions from all the other submissions to
>     the same repo.  To make it easier for reviewers to be notified of
>     submissions without "watching" the whole repository, I suggest
>     that we tweak this process slightly by making step 2 mandatory
>     rather than optional.  (Or, we could talk to the Test Management
>     TF about whether it's better to split the web platform test repo
>     into multiple repos.)
>
>     If the group accepts this process, I'll start migrating our
>     existing test submissions to GitHub by filing a pull request for
>     each one (or by getting the originally submitters to file the pull
>     requests). I'll also work with Dave Methvin to migrate the
>     submissions from his Test The Web Forward repo [7].
>
>     [1]
>     https://github.com/w3c/web-platform-tests/tree/master/pointerevents
>     [2] https://github.com/w3c/web-platform-tests/
>     [3] http://testthewebforward.org/resources/github_test_submission.html
>     [4] http://www.w3.org/wiki/Webapps/Testing
>     [5] http://www.w3.org/wiki/Testing/Test_Management_TF
>     [6]
>     http://lists.w3.org/Archives/Public/public-test-infra/2013AprJun/0066.html
>     [7] https://github.com/dmethvin/pointerevents-test/
>
>

Received on Tuesday, 4 June 2013 13:39:49 UTC