Re: Consolidating css-wg and web-platform-tests repositories (Was: test suite meta data)

On Aug 1, 2013, at 1:10 PM, Tobie Langel wrote:

> On Thursday, August 1, 2013 at 6:37 PM, Linss, Peter wrote:
>> On Jul 31, 2013, at 3:43 PM, Tobie Langel wrote:
>>> What is preventing us from consolidating the CSS and web-platform-test repositories at this stage?
>> 
>> The fact that we have ~2000 review comments over the course of two years in a pre-existing system that we're not willing to throw away. The plan of record is that I'm going to be adapting our existing system to integrate with GitHub.
> 
> I don't understand why switching to the main repo would cause you to loose these two years of review comments. Surely, these could be kept accessible despite the new system being hosted elsewhere.

The comments themselves wouldn't be lost per se, we'd just lose all correlation between the assets in the old repo and the new one making the old data effectively useless. Furthermore, because we have allowed tests to be landed prior to review, everything that isn't currently approved will have to have an issue filed against it to match the new model. The current number is over 4300 tests and references in that state. Hopefully you can see that this is untenable without some automation.

Beyond that, we also have a significant infrastructure built around our current repository, i.e. the build system, test harness, spec annotations, etc. I'm also working on making the build system more generic so it can run against the new repo layout. This has to be completed before we can do any significant rearranging of our files.

> Furthermore, the main repo's review model and test lifecycle models are very different from the CSS WG's: test are approved when pulled in. If a test has a problem, an issue is filed against it. That's all there is to it. No test status/life cycle, etc. just regular versioning. The CSS WG will need to adopt this workflow when switching to the main repo or convince everyone else to adopt the CSS WG's way (which is highly unlikely).

We will be adopting the new workflow when we switch, in fact, we've already adopted it to a large extent by mirroring our repo on GitHub and accepting PR in the same fashion as the main repo does. Part of the changes to Shepherd are to make this workflow work with the existing systems we have.

Also note that historically, we allow un-approved tests into our repository so that they can get included in our test suite builds and evaluated by actually using them in our running test suites without calling them 'approved'. In other words, we haven't blocked all tests waiting for someone to review them. If we did, we'd still be trying to get CSS2.1 out the door.

I'm not convinced that the current model of not merging tests until they're reviewed is going to work for every situation, but that's another discussion.

> I'm not sure what benefits there are in delaying the switch further.

As I said above, the benefit to us is not breaking a while bunch of infrastructure that we've been using for years and are relying on. We have to be able to shift our infrastructure before we can move the tests. In the meanwhile, we're doing what we can to minimize the differences.

>> Once that's working, and we can merge our existing review data, we'll move our tests into the main repo.
> 
> What's the timeline for that?

Sometime this year. I'm actively working on it.

> Keeping a common set of docs and process across the two repos is already proving to be difficult. As we start building more infrastructure around the main repo, I'm concerned we'll quickly get further apart than we're now.

Well, a good way to minimize the drift is to keep me in the loop with the decisions and plans for the infrastructure that's being build around the main repo. That way I can keep our systems inline to the best extent possible.

I'm not sure where the disconnect is, maybe I missed subscribing to a mailing list, or people keep forgetting to invite me to ad-hoc meetings, or something, but I often have the impression that decisions for the new infrastructure are being made in isolation without any kind of public discussion, let alone due consideration for other working group's existing needs. 

I'm sure there are also plans for the new infrastructure that can be achieved more easily by leveraging systems that we've already built rather than starting all over from scratch. I'm more than willing to contribute, but I have to know what those plans are...

Peter

Received on Thursday, 1 August 2013 21:03:05 UTC