W3C home > Mailing lists > Public > public-test-infra@w3.org > April to June 2012

Re: Repository layout

From: James Graham <jgraham@opera.com>
Date: Thu, 31 May 2012 16:15:40 +0200
Message-ID: <4FC77D0C.2090404@opera.com>
To: public-test-infra@w3.org
On 05/31/2012 03:53 PM, Robin Berjon wrote:
> Hi James,
>
> thanks for bringing this up, it so happens that just as you were
> writing this email Dom and I were discussing the exact same problem.
>
> On May 30, 2012, at 13:53 , James Graham wrote:
>> For many test repositories we are using a submitted/approved
>> directory structure. This is not working well for several reasons:
>>
>> * There are typically far more useful tests in the submitted
>> directories than the approved directories. This is due to a general
>> lack of time/interest in reviewing tests (I am just as guilty as
>> anyone). I doubt this situation will change.
>>
>> * It makes it difficult for us to import the tests. Because of the
>> way we test it is very helpful if the paths to tests remain
>> constant (cool URIs and all that). Moving the tests around is a
>> severe inconvenience as we have to update our metadata with the new
>> paths.
>
> Additionally, it is causing regular problems with the Test Framework
> tool. Currently, submitted test suites tend to be imported into it so
> that people can run them, but when tests are moved around the DB
> entries are not updated either for the approved or for the submitted
> suites. Changing this will not make this problem entirely go away
> since it's possible that tests will still be removed, but it ought to
> make this breakage much rarer.

This sounds a lot like the class of problems we encounter.

> The test framework supports flags. We could certainly ask people to
> add a "reviewed" flag when they actually review a test

That sounds ideal to me

> Overall I agree that the approval step is a bureaucratic speed bump
> that is not being helpful. I think that we should move to a
> commit-then-review model in which people who have an interest in
> passing a test suite can file bugs against broken tests. Ideally, we
> would make flagging broken tests easy  I'm thinking about ways of
> doing that in the framework (suggestions welcome  I wonder if I
> could just add a flag to "reported as broken" test cases).

Make it possible to link to bug reports?
Received on Thursday, 31 May 2012 14:16:16 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Thursday, 31 May 2012 14:16:16 GMT