Re: Repository layout

On May 31, 2012, at 2:48 AM, Aryeh Gregor wrote:

> On Wed, May 30, 2012 at 8:46 PM, James Graham <jgraham@opera.com> wrote:
>> I don't think we want a build step. In our ideal scenario, we clone the
>> repository, create a local branch with any patches we need to get things
>> working in our test setup, and then update the testsuite with a simple
>> fetch/rebase (plus whatever work is needed on our end to add any new tests).
>> Everything that requires a QA to set up a build environment and mess around
>> getting the output files from the build into a local repository that's not
>> just a clone of the upstream is an impediment to keeping the tests up to
>> date. We have a lot of experience with testsuites that are hard to update,
>> and it all suggests that the probability of someone doing the update falls
>> dramatically as the complexity of the operation increases.
> 
> Mozilla imports testharness.js tests in a similar fashion -- clone the
> source repo, copy any files indicated in the manifests, and that's it.
> (We don't have a way to maintain local patches, since
> testharnessreport.js serves our need well enough.)  So at least for
> testharness.js tests, I also don't think we need a separate build
> step.

As I said, the submitted/approved distinction in the repository is a carryover. It's not necessary to use if it gets in your way. However, I'm opposed to a change that mandates that it _not_ be used.

The bottom line is that different groups have different testing requirements and we need to accommodate them all in the tools (as best we can). Some tests (like I18n php tests) can't move around and can't use a build step, others, like CSS, _need_ a build step for format conversions.

My point is that if a build step works for your group, then the repo layout is somewhat irrelevant. If you don't want or can't use a build step, then I agree that test assets should not be moving around in the repository. Also note that your repository layout will have consequences to the workflow of people contributing to your suite, if that's a small number and/or it's a small suite, a simple flat structure will most likely work best. If there are a _lot_ of tests[1] and a lot of contributors (many of whom are not software developers familiar with best practices using VCS in a team environment), having a way to keep them from stepping on each other is a good thing.

Peter

[1] the CSS test repository for example, currently has over 13,000 tests, references and support files.

Received on Thursday, 31 May 2012 14:46:32 UTC