Re: Repository layout

On Wed, 30 May 2012, Linss, Peter wrote:

> On May 30, 2012, at 4:53 AM, James Graham wrote:
>
>> For many test repositories we are using a submitted/approved directory structure. This is not working well for several reasons:
>>
>> * There are typically far more useful tests in the submitted directories than the approved directories. This is due to a general lack of time/interest in reviewing tests (I am just as guilty as anyone). I doubt this situation will change.
>>
>> * It makes it difficult for us to import the tests. Because of the way we test it is very helpful if the paths to tests remain constant (cool URIs and all that). Moving the tests around is a severe inconvenience as we have to update our metadata with the new paths.
>>
>> I suggest we go with a single-level directory structure. If people want to keep metadata about which tests have been reviewed that should not be encoded in the filesystem heirachy. I doubt we will do any better than assuming that tests are good and that implementors will file bugs when they find a bad test.
>>
>
>
> The submitted/approved split is a carryover from the CSS test repository 
> before we had a test tracking tool. Even though we do have a tool now, 
> we still find it somewhat useful to have a distinction between 'blessed' 
> tests and those that are being worked on and haven't had any real kind 
> of review.

I haven't yet seen any evidence that this distinction has been useful for 
html/webapps. On the other hand I do see harm it causes as test paths/URLs 
change.

If I thought this kind of distinction was useful I would suggest 
implementing it at the VCS level rather than at the filesystem level.

> The expectation when other tools import the tests is that they not 
> import directly from the source tree, but from the build output. The CSS 
> test suites have build tools that gather all the tests in the various 
> source directories and copy the proper tests into per-suite output 
> directories. The build tools also generate manifest files for other 
> systems to import the test data (and metadata) as well as human readable 
> index files.
>
> I'm working on making the CSS build tools more generic so they can be 
> used with other test suites. Once that's done, if you use the build 
> output then the layout of the source repository is irrelevant to your 
> tools.

I don't think we want a build step. In our ideal scenario, we clone the 
repository, create a local branch with any patches we need to get things 
working in our test setup, and then update the testsuite with a simple 
fetch/rebase (plus whatever work is needed on our end to add any new 
tests). Everything that requires a QA to set up a build environment and 
mess around getting the output files from the build into a local 
repository that's not just a clone of the upstream is an impediment to 
keeping the tests up to date. We have a lot of experience with testsuites 
that are hard to update, and it all suggests that the probability of 
someone doing the update falls dramatically as the complexity of the 
operation increases.

Received on Wednesday, 30 May 2012 17:47:13 UTC