Re: test suite meta data

On 7/31/13 3:13 PM, "Tobie Langel" <tobie@w3.org> wrote:

>On Wednesday, July 31, 2013 at 9:29 PM, James Graham wrote:
>> My point is that a specific review can often generate dozens of mails
>> (in no small part due to the way that github insists on one mail per
>> comment). Getting all these mails for a review that you are not
>> interested in is tedious, even if you are interested in other reviews
>> for the same spec. I don't see how it is possible to opt-out of these
>> mails on a per-review basis in the proposed system. Based on experience
>> of review systems in general - even less spammy ones than github -
>> people will quickly become annoyed by this.
>> 
>> It is possible that I misunderstood and that tobie is proposing one
>> email per review to the mailing list, which is OK, but does make it
>>hard 
>> to follow activity on reviews you do care about (I don't know, but I
>> guess it must be possible to subscribe on a per-review basis using
>> github alone?).
>
>Thanks for your insightful comments, James. This brings up a number of
>issues I hadn't completely thought through.
>
>Here's my full proposal:
>
>1. Folks should set their notification status to "not watching" (you only
>get mails when you're mentioned explicitly in a comment, you can easily
>mute the thread if you want).
>2. When a new PR arrives we look at which files it touches and tag it
>accordingly.
>3. We then go look in the relevant directory for the manifest.
>-> if there's a specified test contributor, the PR is automatically
>assigned to him,
>-> if there's a list of reviewers, a comment is added to the PR with
>their handles, that'll notify them personally.
>-> if there's a mailing list present, an email is sent to that mailing
>list prefixed with [shortname-tests].
>-> an email is sent to web-platform-tests-notifications@w3.org for
>archiving purposes (or for those who want to drink from the fire-hose).
>Also prefixed with [shortname-tests].
>3. When further comments, new commits etc. are made to the PR,
>-> we notify web-platform-tests-notifications@w3.org
>-> we notify the relevant mailing lists if they've opted in to get that
>type of notification.
>
>This makes the manifest now look something like this:
>
>{
>    "test-coordinator": "KrisKrueger",
>    "test-reviewers": ["KrisKrueger", "jgraham", "Ms2ger"],
>    "mailing-list": [
>        {
>            "address": "public-webapps-testsuite@w3.org"
>            "comments": false,
>            "commits": true
>        }
>    ]
>
>} 
>
>Would that address your concerns?

This all looks great, but I'm wondering/hoping if this is code that can be
run on the csswg-test repo as well.  It's proving to be super-complicated
trying to document both worlds. Even if it takes a bit more time to
consolidate the repositories, it would be great if they ran mostly the
same way in parallel in the interim.  Peter Linss, can you comment on this?



>>> On a somewhat related note, right now I happen to be editing the doc
>> > that
>> > describes how to notify people of new test submissions [2]. It
>> > suggests
>> > (optionally) notifying public-test-infra or #testing in IRC (as
>>listed 
>> > in
>> > the Communication Channels doc [3]). Should this be changed?
>> 
>> Well I at least currently get multiple emails per PR, so one more to
>> public-test-infra isn't that helpful.
>
>Agreed. We don't want contributors to ping reviewers more than necessary.
>Let's strike this.

Ok, I'll change:

This will notify all subscribers of the PR. The review initiator can also
notify the group of the PR via the mailing list in our communication
channels.

To: 
This will notify all subscribers of the PR, including the test coordinator
and test reviewers.




> 
>> Asking questions on #testing is
>> fine of course.
>
>+1 
>
>--tobie

Received on Wednesday, 31 July 2013 22:35:46 UTC