W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2012

Re: Repository layout

From: Robin Berjon <robin@berjon.com>
Date: Thu, 31 May 2012 15:53:38 +0200
Cc: public-test-infra@w3.org
Message-Id: <DFE14D99-466B-4351-935E-B0C5A3D7E25A@berjon.com>
To: James Graham <jgraham@opera.com>
Hi James,

thanks for bringing this up, it so happens that just as you were writing this email Dom and I were discussing the exact same problem.

On May 30, 2012, at 13:53 , James Graham wrote:
> For many test repositories we are using a submitted/approved directory structure. This is not working well for several reasons:
> 
> * There are typically far more useful tests in the submitted directories than the approved directories. This is due to a general lack of time/interest in reviewing tests (I am just as guilty as anyone). I doubt this situation will change.
> 
> * It makes it difficult for us to import the tests. Because of the way we test it is very helpful if the paths to tests remain constant (cool URIs and all that). Moving the tests around is a severe inconvenience as we have to update our metadata with the new paths.

Additionally, it is causing regular problems with the Test Framework tool. Currently, submitted test suites tend to be imported into it so that people can run them, but when tests are moved around the DB entries are not updated either for the approved or for the submitted suites. Changing this will not make this problem entirely go away since it's possible that tests will still be removed, but it ought to make this breakage much rarer.

What's more, submitted tests are inside ./submitted/$submitter/foo.html and approved tests are in ./approved/foo.html  a lot of the time they reference ../../common/some-useful-script.js and break when you move them. It's not a huge deal, but it's a regular annoyance.

> I suggest we go with a single-level directory structure. If people want to keep metadata about which tests have been reviewed that should not be encoded in the filesystem heirachy. I doubt we will do any better than assuming that tests are good and that implementors will file bugs when they find a bad test.

The test framework supports flags. We could certainly ask people to add a "reviewed" flag when they actually review a test (alternatively we can detect this with the <link rel=reviewer> convention). That should provide us with enough information to do whatever we need to do, e.g. generate a suite run containing only reviewed tests if someone needs that for some reason.

Overall I agree that the approval step is a bureaucratic speed bump that is not being helpful. I think that we should move to a commit-then-review model in which people who have an interest in passing a test suite can file bugs against broken tests. Ideally, we would make flagging broken tests easy  I'm thinking about ways of doing that in the framework (suggestions welcome  I wonder if I could just add a flag to "reported as broken" test cases).

-- 
Robin Berjon - http://berjon.com/ - @robinberjon
Received on Thursday, 31 May 2012 13:54:10 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 31 May 2012 13:54:11 GMT