RE: Repository layout

Response inline.

Thanks,
Bryan Sullivan 

-----Original Message-----
From: James Graham [mailto:jgraham@opera.com] 
Sent: Wednesday, May 30, 2012 4:53 AM
To: public-test-infra@w3.org
Subject: Repository layout

For many test repositories we are using a submitted/approved directory 
structure. This is not working well for several reasons:

* There are typically far more useful tests in the submitted directories 
than the approved directories. This is due to a general lack of 
time/interest in reviewing tests (I am just as guilty as anyone). I 
doubt this situation will change.

* It makes it difficult for us to import the tests. Because of the way 
we test it is very helpful if the paths to tests remain constant (cool 
URIs and all that). Moving the tests around is a severe inconvenience as 
we have to update our metadata with the new paths.

I suggest we go with a single-level directory structure. If people want 
to keep metadata about which tests have been reviewed that should not be 
encoded in the filesystem heirachy. I doubt we will do any better than 
assuming that tests are good and that implementors will file bugs when 
they find a bad test.

<bryan> I like the idea of using metadata to track which tests have been approved. The test framework could then offer the option of running a specific test, all submitted tests, or only approved ones. This would enable the review process by making it easier to execute tests and provide feedback directly through the test framework (e.g. when running a test singly, after the test executes the tester could rate the test quality, and that can be captured along with the test result).

Received on Thursday, 31 May 2012 23:39:05 UTC