Re: Repository layout

On Jun 1, 2012, at 15:49 , Linss, Peter wrote:
> On Jun 1, 2012, at 5:13 AM, Robin Berjon wrote:
>> I know, but the problem is that since people are treating the file system layout as a meaningful-enough classification for tests, they're not bothering to update the manifest and resubmitting it. Ideally the metadata would be intrinsic to the test (externalised, authoritative metadata is almost always a bad idea) and the framework would automatically know of updates. Otherwise things are bound to get out of synch.
> 
> Well, that's the way the system was designed. If people don't update the inputs to the system it will always have bad data. 

Well obviously. But as we scale these tools we have to take the social component into account. If we notice that people aren't updating the useful metadata, then we ought to make sure that the tools compensate for that — it's a lot easier to patch tools than humans (in my experience at least ;).

>> No, they're definitely not, because most test suites do not require a build step.
> 
> Right, but eventually they should be using the build system to produce the manifests at least. For small suites that don't change often, manifests could be produced by hand, but that gets unmaintainable fast.

Sure, but if all the build step does is read the test files in order to produce the manifest, why not have the tool that reads the manifest read the test files directly, assuming a common metadata format (the foundations for which we have) and common rules for finding tests in a directory tree? I know Shepherd handles this — I'm mostly wondering about orthogonalising all of that and limiting the amount of integration between tools and various moving parts that we need to all be happy.

>> I'm aware of Shepherd, but I've only seen it in use for CSS. It's hard to know if it is well suited for other suites or not with just that view. With that in mind it would be a good idea to integrate it on w3c-test.org to see if it applies well.
> 
> That's the plan. It was designed to be generic and configurable. It's still going through enough development that the DB schema is still getting adjusted, so while we're using it for production use, I'm not sure it's stable enough to be setup on w3c-test.org as I don't have enough access to that box to keep it in sync with development. We should talk about that more in a few weeks though…

That's part of what worries me. Maintaining a single DB schema for all the varied needs that groups seem to have may require it to shoulder more weight than would be easy to work with. We can make this easier to handle using Doctrine migrations (or some similar solution, maybe Symfony Components have something for this nowadays) but it still requires a lot of integration. I'm wondering if a more "unixish" approach with small tools tied together in ad hoc manners would not be simpler for everyone involved (and allow us all to make progress without waiting on one another's updates).

> Another point. Both Shepherd and the framework were designed to have installs mapped one-to-one with test repositories (especially Shepherd, the frame work is more flexible but there's still the test name uniqueness issue).

Not to come across as a dangerously radical anarchist, but could we perhaps identify tests with URIs? Hell, URLs even!

> I don't think every group should be dumping all their test suites into a single install. It won't scale to cover the entire W3C with a single instance. They deployment plan needs to be mapped out better before it gets out of hand. I consider the current install of the framework on w3c-test to be more experimental/evaluation than final.

Ok, that's a very useful data point because I think it is at odds with the expectations in developing the w3c-test setup (I'm not blaming you in any way, just pointing out the lack of alignment here :). If Shepherd isn't going to scale to the usage we'd intended, we definitely need a rethink.

Thanks!

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Monday, 4 June 2012 17:26:46 UTC