Re: Repository layout

On May 31, 2012, at 17:26 , Linss, Peter wrote:
> On May 31, 2012, at 6:53 AM, Robin Berjon wrote:
>> Additionally, it is causing regular problems with the Test Framework tool. Currently, submitted test suites tend to be imported into it so that people can run them, but when tests are moved around the DB entries are not updated either for the approved or for the submitted suites. Changing this will not make this problem entirely go away since it's possible that tests will still be removed, but it ought to make this breakage much rarer.
> 
> My guess here is that the manifests are probably not being used properly (though I haven't really looked at the particular problem you're describing). The test case import should be removing tests not found in a newer version of the manifest from the suite, although the tests themselves and their results will be preserved in the DB (by design). This way if the test shows up in a different suite (or the same one later) the old results are still there.

I know, but the problem is that since people are treating the file system layout as a meaningful-enough classification for tests, they're not bothering to update the manifest and resubmitting it. Ideally the metadata would be intrinsic to the test (externalised, authoritative metadata is almost always a bad idea) and the framework would automatically know of updates. Otherwise things are bound to get out of synch.

> My understanding is that most of the suites aren't using the revision information for the tests properly either. This is another field that the build system populates properly when it generates the manifests.

No, they're definitely not, because most test suites do not require a build step.

> Even for those test suites where a build step isn't necessary, the build tools can still be useful for generating the proper manifest files...

But you still need to generate them, and upload them. We're not getting enough tests and enough reviews as it is, every step that is added — no matter how small — decreases our success rate.

>> The test framework supports flags. We could certainly ask people to add a "reviewed" flag when they actually review a test (alternatively we can detect this with the <link rel=reviewer> convention). That should provide us with enough information to do whatever we need to do, e.g. generate a suite run containing only reviewed tests if someone needs that for some reason.
> 
> Please take a look at Shepherd[1], it already uses the <link rel=reviewer> metadata to mark tests as approved.

I'm aware of Shepherd, but I've only seen it in use for CSS. It's hard to know if it is well suited for other suites or not with just that view. With that in mind it would be a good idea to integrate it on w3c-test.org to see if it applies well.

>> Overall I agree that the approval step is a bureaucratic speed bump that is not being helpful. I think that we should move to a commit-then-review model in which people who have an interest in passing a test suite can file bugs against broken tests. Ideally, we would make flagging broken tests easy — I'm thinking about ways of doing that in the framework (suggestions welcome — I wonder if I could just add a flag to "reported as broken" test cases).
> 
> The framework already has the support for that, though no UI (yet). A possible result is 'invalid', which marks the test as bad until it is modified.

Ah, good. I've seen invalid in the source but couldn't figure out what it did.

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Friday, 1 June 2012 12:16:25 UTC