Re: CSS test GitHub repo strategy for TestTWF Shenzhen

On Oct 3, 2013, at 4:52 PM, Tobie Langel wrote:

> On Thursday, October 3, 2013 at 4:58 PM, Rebecca Hauck wrote:
>> On 10/3/13 7:25 AM, "Tobie Langel" <tobie@w3.org (mailto:tobie@w3.org)> wrote:
>>> On a branding/discoverability perspective, I'm also not keen on seeing a
>>> testthewebforward org pop up on GH that wouldn't also be the canonical
>>> location of the testtwf-website repo, web-platform-tests repo, etc. As
>>> moving those repos would just be a major annoyance to everyone involved,
>>> I'm strongly opposed to this. If you absolutely must create a new org for
>>> this, please tie it to the CSS WG and not to Test the Web Forward.
>> 
>> Fair enough. FWIW, we created the testhewebforward github user in April in
>> preparation for the Seattle event, where we first used GitHub. I only just
>> converted it to an organization so I could add more owners. I'm aiming for
>> this to be a repeatable process for other events and other organizers who
>> may not be CSSWG members, so I'm not sure a CSSWG organization is the way
>> to go. Would TestTheWebForward-Events be more acceptable?
> 
> No, because that again makes a special case (the CSS WG's reliance on HG as a canonical repository) the perceived norm. One of the key guidelines of this testing effort is discoverability. The existence of a TestTheWebForward branded repo that isn't the canonical location of the tests completely trumps that.
> 
> I have another proposal to make. It is probably a bit crazy, there are obvious downsides to it which might end up being deal breakers, but it elegantly solves all of the problems you've been bringing up here, and more.
> 
> What if we immediately started to use the main repository for contributions of CSS tests made during TestTWF events?
> 
> We could add directories for each CSS WG spec much like we do for the specs of other groups. When the time comes to merge the content of the CSS repo into the main directory, everything would already be setup. In the unfortunate event that the CSS WG would decide not to join the common effort, we could always merge the CSS tests back to the CSS WG's repo at relatively low engineering costs.
> 
> This solution obviously completely solves your workflow issues. It also solves the CLA problem I raised and the issues related to creating a dedicated GH org I mentioned above.
> 
> An added benefit is that it really simplifies the documentation and the explanation done at TestTWF events and on testthewebforward.org. There is one repo, one workflow, one way to do things, etc. It's also nicely forward-looking.
> 
> Now as, I mentioned before, there are obvious downsides to this (and maybe also less obvious ones), but the tradeoff might be worth it. Should we seriously consider this, collect the downsides, and have an informed conversation about it or are there immediate blockers that come to mind and make the exercise moot? 

I think the downsides of this approach make it a non-starter. Doing this would leave us in one of two possible situations:
1) The CSS tests are split across two non-related repositories. 
2) We move all the CSS tests from the WPT repo in to the CSS repo after the event.

#1 is a situation I wouldn't wish upon my worst enemy. Firstly, trying to manage synchronization between two repos will be a nightmare, test suites will get split, tests will get out of sync, and no one will be able to unravel which is the proper state of the combined tests. Second, the CSS test suites are not the contents of the repository, they are the product of a build process, modifying our build process to handle source content from multiple repos will be problematic to say the least (primarily due to the synchronization issues listed previously).

#2 breaks the relationship between the author's original PR and the test source. Future modifications to the original PR will be disconnected form the tests and lost (or will resurrect the test in the WPT repo, re-introducing the sync issues), and to the author it will appear that their submissions have simply disappeared. Not a good way to encourage future contributions.

The only way out of either of the above situations would be to move the CSS tests en-masse into the WPT repo prior to the event, which is something we're not willing to do at this point in time. We'd loose all of our existing review information (or at the least, disconnect it from the repo making the job of ever getting it back in sync a nightmare) and the WPT tooling and process is not yet capable of handling the CSS WG's testing needs (and the question of if it ever will has yet to be settled).

Tobie, I accept your objections to having a TestTWF fork of any of the repos, for all the reasons you outlined. There is another alternative, however, to just living with the status quo. 

The simplest solution here is to simply turn on write access to the CSS WG GitHub repo for the duration of the event (and possibly just leaving it on after). This way the test suite owners can simply merge PRs during the event directly into the GitHub repo and everything will work as expected. I can easily reverse the direction of the Git<->Hg bridge and we just treat the GitHub repo as canonical during the event.

After the event, we can manually re-sync the Hg repo to the GitHub repo if there are any issues with the automatic conversion (or if there are any contributions to the Hg repo during the event). I've also been exploring setting up the Gut<->Hg bridge as bi-directional which would let us leave write access on in both places.

As far as using the CSS WG GitHub issue tracker, that's already been turned on. There are issues with being able to set milestones due to odd GitHub access restrictions, but those go away when write-access is turned on. I'll deal with synchronizing the issues with the Shepherd data.

This leaves the CLA issue, but I can work on either modifying the WPT CLA tool to work with the CSSWG repo, or replicating the functionality via our existing tooling prior to the event.

The only other issue this would leave open is having contributors submit tests to two repos, although the process would be identical so the burden would be minimal. This unfortunately isn't something we can resolve until such time as we're ready to merge the repositories.

Peter