Re: CSS test GitHub repo strategy for TestTWF Shenzhen

On Oct 15, 2013, at 11:38 AM, Rebecca Hauck wrote:

> Hey Peter,
> 
> Following up on this thread because we're just a few weeks from TestTWF
> and there are a few details I'd like to get confirmed.
> 
> 1. We'll be moving forward with your proposal to make the Github repo the
> canonical repo, yes?

Yes.

> 
> 2. When is the earliest possible that this can be ready? May we get it
> switched a week or two (preferably) before the event?

I'm working on adding the bi-directional sync at the moment. I hope to have it running by the end of the week, maybe early next week. (depending on glitches I've been finding in the hg-git code).

Push come to shove I can punt on my current approach and simply reverse the sync for the event fairly quickly, I'm just trying to do it "the right way" first.

> 
> 3. May we use the issue tracker to start logging tests we'll be suggesting
> for people to write?  I'd need permission on the repo to use labels on
> them. I've already got a list accumulating for 3-4 different CSS specs so
> filtering by labels is highly desired.

That's fine by me, but either I'm not finding the GitHub UI to add you to the csswg-test repo or I don't have access to add you. Someone with owner rights in the w3c organization will have to add you to the 'csswg-owners' team. Tobie?

Please don't do any merges or commits directly to the GitHub repo though until I tell you that the sync is in place.

> 4. I assume we'll have to grant certain people the right permissions for
> merging, labeling, etc?  I currently have 4-5 people who'll be supporting
> CSS test groups + some additional CSSWG members confirmed as experts.
> If/when you grant me the proper permissions, will I be able to do this
> myself or do you need a list of github Ids?

It doesn't look like I can grant those permissions at the moment (unless my access gets bumped), it looks like you'll have to get someone with w3c owner rights to either make you an owner, or add the people you want. But again, please don't add anyone else until the sync is set up, having too many people with push access before the sync is ready is just asking for trouble.

> 
> 5. Finally, and this is probably more of a feature request: For this event
> and going forward, we will be adopting the process of the WPT repo in that
> merged PRs will be considered approved. When they land in Shepherd, they
> go in as 'Submitted for Review' if they don't have reviewer links,
> 'Accepted' if reviewed by someone not on a short list, or 'Approved' if
> reviewed by Certain People. Since the reviewer is identified in Github,
> it's kind of make-work to have to manually add this as metadata (although
> I realize Shepherd needs it there). Is there a way automate this and
> insert the reviewer info base on the person who does the merge?

There isn't currently. We can add some micro syntax to the merge, something like the person doing the merge adds '[approved]' to the beginning of the commit message when merging. Or the person doing the merge can just set the status to approved in Shepherd, you don't have to have the metadata in the test.

> 
> Also, for this workflow, I'd like to promote the TestTWF experts in
> Shepherd to be on the list of Certain People whose reviewed tests are
> automatically set to Approved.  It will cut down the work I have to do in
> Shepherd afterward. Any objection to this?

No objection. You can add approvers in Shepherd yourself in the repository manager UI:
https://test.csswg.org/shepherd/manager/roles/

Just add the people you want to the 'testing-peer' group.

> 
> Prior to the event or when you get these changes in place, please let me
> know if there are any details about this workflow that might be impactful
> during or after the event.  I'm happy to meet or do a call with you
> anytime.
> 
> Thanks,
> -Rebecca
> 
> 
> On 10/4/13 1:01 PM, "Rebecca Hauck" <rhauck@adobe.com> wrote:
> 
>> 
>> 
>> On 10/4/13 11:48 AM, "Peter Linss" <peter.linss@hp.com> wrote:
>> 
>>> 
>>> On Oct 3, 2013, at 4:52 PM, Tobie Langel wrote:
>>> 
>>>> On Thursday, October 3, 2013 at 4:58 PM, Rebecca Hauck wrote:
>>>>> On 10/3/13 7:25 AM, "Tobie Langel" <tobie@w3.org
>>>>> (mailto:tobie@w3.org)> wrote:
>>>>>> On a branding/discoverability perspective, I'm also not keen on
>>>>>> seeing a
>>>>>> testthewebforward org pop up on GH that wouldn't also be the
>>>>>> canonical
>>>>>> location of the testtwf-website repo, web-platform-tests repo, etc.
>>>>>> As
>>>>>> moving those repos would just be a major annoyance to everyone
>>>>>> involved,
>>>>>> I'm strongly opposed to this. If you absolutely must create a new org
>>>>>> for
>>>>>> this, please tie it to the CSS WG and not to Test the Web Forward.
>>>>> 
>>>>> Fair enough. FWIW, we created the testhewebforward github user in
>>>>> April in
>>>>> preparation for the Seattle event, where we first used GitHub. I only
>>>>> just
>>>>> converted it to an organization so I could add more owners. I'm aiming
>>>>> for
>>>>> this to be a repeatable process for other events and other organizers
>>>>> who
>>>>> may not be CSSWG members, so I'm not sure a CSSWG organization is the
>>>>> way
>>>>> to go. Would TestTheWebForward-Events be more acceptable?
>>>> 
>>>> No, because that again makes a special case (the CSS WG's reliance on
>>>> HG as a canonical repository) the perceived norm. One of the key
>>>> guidelines of this testing effort is discoverability. The existence of a
>>>> TestTheWebForward branded repo that isn't the canonical location of the
>>>> tests completely trumps that.
>>>> 
>>>> I have another proposal to make. It is probably a bit crazy, there are
>>>> obvious downsides to it which might end up being deal breakers, but it
>>>> elegantly solves all of the problems you've been bringing up here, and
>>>> more.
>>>> 
>>>> What if we immediately started to use the main repository for
>>>> contributions of CSS tests made during TestTWF events?
>>>> 
>>>> We could add directories for each CSS WG spec much like we do for the
>>>> specs of other groups. When the time comes to merge the content of the
>>>> CSS repo into the main directory, everything would already be setup. In
>>>> the unfortunate event that the CSS WG would decide not to join the
>>>> common effort, we could always merge the CSS tests back to the CSS WG's
>>>> repo at relatively low engineering costs.
>>>> 
>>>> This solution obviously completely solves your workflow issues. It also
>>>> solves the CLA problem I raised and the issues related to creating a
>>>> dedicated GH org I mentioned above.
>>>> 
>>>> An added benefit is that it really simplifies the documentation and the
>>>> explanation done at TestTWF events and on testthewebforward.org. There
>>>> is one repo, one workflow, one way to do things, etc. It's also nicely
>>>> forward-looking.
>>>> 
>>>> Now as, I mentioned before, there are obvious downsides to this (and
>>>> maybe also less obvious ones), but the tradeoff might be worth it.
>>>> Should we seriously consider this, collect the downsides, and have an
>>>> informed conversation about it or are there immediate blockers that come
>>>> to mind and make the exercise moot?
>>> 
>>> I think the downsides of this approach make it a non-starter. Doing this
>>> would leave us in one of two possible situations:
>>> 1) The CSS tests are split across two non-related repositories.
>>> 2) We move all the CSS tests from the WPT repo in to the CSS repo after
>>> the event.
>>> 
>>> #1 is a situation I wouldn't wish upon my worst enemy. Firstly, trying to
>>> manage synchronization between two repos will be a nightmare, test suites
>>> will get split, tests will get out of sync, and no one will be able to
>>> unravel which is the proper state of the combined tests. Second, the CSS
>>> test suites are not the contents of the repository, they are the product
>>> of a build process, modifying our build process to handle source content
>>> from multiple repos will be problematic to say the least (primarily due
>>> to the synchronization issues listed previously).
>>> 
>>> #2 breaks the relationship between the author's original PR and the test
>>> source. Future modifications to the original PR will be disconnected form
>>> the tests and lost (or will resurrect the test in the WPT repo,
>>> re-introducing the sync issues), and to the author it will appear that
>>> their submissions have simply disappeared. Not a good way to encourage
>>> future contributions.
>>> 
>>> The only way out of either of the above situations would be to move the
>>> CSS tests en-masse into the WPT repo prior to the event, which is
>>> something we're not willing to do at this point in time. We'd loose all
>>> of our existing review information (or at the least, disconnect it from
>>> the repo making the job of ever getting it back in sync a nightmare) and
>>> the WPT tooling and process is not yet capable of handling the CSS WG's
>>> testing needs (and the question of if it ever will has yet to be
>>> settled).
>>> 
>>> Tobie, I accept your objections to having a TestTWF fork of any of the
>>> repos, for all the reasons you outlined. There is another alternative,
>>> however, to just living with the status quo.
>>> 
>>> The simplest solution here is to simply turn on write access to the CSS
>>> WG GitHub repo for the duration of the event (and possibly just leaving
>>> it on after). This way the test suite owners can simply merge PRs during
>>> the event directly into the GitHub repo and everything will work as
>>> expected. 
>> 
>> Yes!! I didn't realize that was an option, but if this constraint were
>> lifted, it will solve the bulk of the problems I'm trying to avoid. If
>> you're willing to proceed with this, I'd ask that we get a little lead
>> time of at least a few days so we can make sure it's all working properly
>> by the day of the event.  Also, can/should we employ a branching strategy
>> here? I.e., create at testtwf-shenzhen branch and merge PRs into that.  I
>> don't know if make things more or less complicated, but it seems like a
>> good use of branching.
>> 
>> 
>> 
>>> I can easily reverse the direction of the Git<->Hg bridge and we just
>>> treat the GitHub repo as canonical during the event.
>> 
>> What are the implications of the work to do afterward? Implicit in my
>> wishes to reduce the pain during the event is to also reduce mine
>> personally afterward. I'm willing to do some follow-up work if necessary,
>> but I'd like to get an idea of what it might be.
>> 
>>> 
>>> After the event, we can manually re-sync the Hg repo to the GitHub repo
>>> if there are any issues with the automatic conversion (or if there are
>>> any contributions to the Hg repo during the event). I've also been
>>> exploring setting up the Gut<->Hg bridge as bi-directional which would
>>> let us leave write access on in both places.
>> 
>> So if a manual re-sync is only required if something goes wrong or if
>> someone contributes to the HG repo in the interim, can we just make HG
>> read-only for this period of time to control that part of it?  A public
>> announcement and a pointer to the Github workflow should cover that.
>> 
>>> 
>>> As far as using the CSS WG GitHub issue tracker, that's already been
>>> turned on. There are issues with being able to set milestones due to odd
>>> GitHub access restrictions, but those go away when write-access is turned
>>> on. I'll deal with synchronizing the issues with the Shepherd data.
>> 
>> Labels are really what I'm after, but I think they're lumped in with
>> milestones in the access restrictions you're talking about. I haven't yet
>> decided if or how I'd want to use milestones, but please let me know if
>> using them in some way interferes with something you're working on. For
>> example, I see that most/all of the CSS specs are in there now as
>> milestones. Should I stay away from those? Is it ok to create new ones?
>> 
>> 
>>> 
>>> This leaves the CLA issue, but I can work on either modifying the WPT CLA
>>> tool to work with the CSSWG repo, or replicating the functionality via
>>> our existing tooling prior to the event.
>> 
>> I believe I already took care of this. [1]
>> 
>>> 
>>> The only other issue this would leave open is having contributors submit
>>> tests to two repos, although the process would be identical so the burden
>>> would be minimal. This unfortunately isn't something we can resolve until
>>> such time as we're ready to merge the repositories.
>> 
>> The alternative you're proposing here solves the majority of my concerns
>> so that we can live with this a while longer.
>> 
>> Thanks so much, Peter!
>> 
>> -Rebecca
>> 
>> 
>> 
>> [1] 
>> https://github.com/w3c/csswg-test/commit/4a3a6280989d62aeac6bb96b09fa9e0d4
>> 5
>> bfedc8
>> 
>> 
>