Re: Notes on test suite reorganisation

On 12/11/2012 01:03 PM, Robin Berjon wrote:
> Hi all!
>
> We've been talking about test suite reorganisation for a while and I
> thought it might be helpful if I put a concrete proposal out there so
> that we can all see what sticks and what doesn't, and hopefully come to
> some conclusion shortly.

This looks like a great start, thanks!

> • Inside tests, I made five directories: html5, html51, canvas2d,
> canvas2d2, microdata to reflect the various specs. Technically there
> might be a microdata2 as well, but there doesn't seem to be much motion
> there for now so that can wait.

I think that HTML5 vs HTML51 is a bad distinction to make in the 
filesystem (see the other branch of this thread).

> Another big topic is how to handle submissions and approved tests. There
> are several options:
>
>      A) Use approved and submissions/Foo subdirectories
>      B) Use pull requests
>      C) Use a file that lists what's approved and what isn't
>
> I think we should rule option (A) out outright. Tests should be moved
> around as little as possible, ideally never.

Agreed.

> Option (B) is interesting, but my concern is getting a view of the
> entire set of submissions + approved tests. We *could* use the GH API to
> obtain the full list of pending PRs and extract the content accordingly,
> but that introduces reliance on the GH API beyond just git (which may
> not be a problem given http://gitlabhq.com/).

Let's restate Option (B) as "use branches, one per submission (note: not 
one per file)". This then frees us from the idea that this option is 
particularly tied to github; pull requests are effectively just branches 
with a fancy name.

The major advantage of this approach is that good code review tools (and 
even mediocre code review tools like the one built in to github) are 
commonly organised around the paradigm that a branch is the unit of code 
review. As issues are identified in the submission, you keep pushing 
fixes to the branch until you fixed all the problems, then you 
integrate. Indeed if we want git-based code review (rather than 
something custom like the CSSWG has) I think we *have* to use this 
option to some extent.

The net result will be that the approved tests are obvious — they are 
just mainline. The list of unapproved submissions is obvious; it's just 
the list of all open review requests. The only problem is that it's hard 
to run all approved + unapproved tests at once because that means 
creating a new branch that includes all the commits from the review 
branches. I suppose it isn't impossible to imagine automating the 
creation of such a thing, but there will inevitably be problems from 
merge conflicts and so on.


> Option (C) is the simplest, though it runs the risk of someone
> forgetting to update the file listing the approved tests (this could
> however be made more obvious by listing content on both sides clearly,
> and possibly spamming this group with "pending submissions" every week).
>
> Overall I have a slight preference for (C), but I could be convinced to
> go with (B), especially if integration with epic (or whatever) is
> particularly good there.

Did you mean Critic? Certainly critic requires that you work that way.

Received on Tuesday, 11 December 2012 13:54:38 UTC