Re: test suite meta data

On Wednesday, July 31, 2013 at 9:29 PM, James Graham wrote:
> My point is that a specific review can often generate dozens of mails
> (in no small part due to the way that github insists on one mail per 
> comment). Getting all these mails for a review that you are not 
> interested in is tedious, even if you are interested in other reviews 
> for the same spec. I don't see how it is possible to opt-out of these 
> mails on a per-review basis in the proposed system. Based on experience 
> of review systems in general - even less spammy ones than github - 
> people will quickly become annoyed by this.
> 
> It is possible that I misunderstood and that tobie is proposing one 
> email per review to the mailing list, which is OK, but does make it hard 
> to follow activity on reviews you do care about (I don't know, but I 
> guess it must be possible to subscribe on a per-review basis using 
> github alone?).

Thanks for your insightful comments, James. This brings up a number of issues I hadn't completely thought through.

Here's my full proposal:

1. Folks should set their notification status to "not watching" (you only get mails when you're mentioned explicitly in a comment, you can easily mute the thread if you want).
2. When a new PR arrives we look at which files it touches and tag it accordingly.
3. We then go look in the relevant directory for the manifest.
-> if there's a specified test contributor, the PR is automatically assigned to him,
-> if there's a list of reviewers, a comment is added to the PR with their handles, that'll notify them personally.
-> if there's a mailing list present, an email is sent to that mailing list prefixed with [shortname-tests].
-> an email is sent to web-platform-tests-notifications@w3.org for archiving purposes (or for those who want to drink from the fire-hose). Also prefixed with [shortname-tests].
3. When further comments, new commits etc. are made to the PR,
-> we notify web-platform-tests-notifications@w3.org
-> we notify the relevant mailing lists if they've opted in to get that type of notification.

This makes the manifest now look something like this:

{
    "test-coordinator": "KrisKrueger",
    "test-reviewers": ["KrisKrueger", "jgraham", "Ms2ger"],
    "mailing-list": [
        {
            "address": "public-webapps-testsuite@w3.org"
            "comments": false,
            "commits": true
        }
    ]

} 

Would that address your concerns?
> > On a somewhat related note, right now I happen to be editing the doc 
> > that
> > describes how to notify people of new test submissions [2]. It 
> > suggests
> > (optionally) notifying public-test-infra or #testing in IRC (as listed 
> > in
> > the Communication Channels doc [3]). Should this be changed?
> 
> Well I at least currently get multiple emails per PR, so one more to 
> public-test-infra isn't that helpful.

Agreed. We don't want contributors to ping reviewers more than necessary. Let's strike this. 
> Asking questions on #testing is 
> fine of course.

+1 

--tobie

Received on Wednesday, 31 July 2013 22:13:31 UTC