- From: James Graham <jgraham@opera.com>
- Date: Tue, 10 May 2011 22:20:17 +0200 (CEST)
- To: "Linss, Peter" <peter.linss@hp.com>
- cc: James Graham <jgraham@opera.com>, public-test-infra@w3.org
On Tue, 10 May 2011, Linss, Peter wrote: > > On May 10, 2011, at 9:15 AM, Linss, Peter wrote: > >> The CSS test repository contains files other than tests as well. Our test harness is not in the test repository, but we do have: >> * files referenced from tests, ie: images and stylesheets >> * reference pages (and files referenced from reference pages) >> * scripts that generate tests >> * apache server configuration files >> * the test suite build code >> >> So yes, Shepherd will be managing and providing a review system for >> files other than tests as well. It will do more with a test file, like >> handle the test metadata, but it will have to handle other files as >> well. > > Another thing Shepherd has to manage is the relationship between tests > and their supporting files. > > For example, the way we organize our repository is that we have separate > directories for tests in development, submitted for review, and approved > tests. When a test is approved, it gets moved from the submitted > directory to the approved directory (this allows workflows by > interacting directly with the repository), when this happens, we need to > make sure all the supporting files get moved into the appropriate place > as well. > > Another case is revision management, the harness tracks revision data > and stores results against the revision of the test. When tests change, > old results are automatically disregarded. The harness needs to know > when a linked stylesheet, image or reference page has changed, even if > the test that uses it didn't. > > For the most part, we're trying to do this by getting the relationship > data directly from the test (e.g. parsing the src of an <img> tag), but > there are cases when the test can't refer the the necessary support > file, like a fail image that only shows up if the browser is computing > relative paths wrong. So we need a mechanism for declaring and tracking > these relationships as well. This is where I think our visions differ. I think that building a good, or even good-enough, review system is enough of a challenge on its own without adding a huge range of testcase management features on top of it. I expect managing tests to be a *hard* problem. For example tests can be identified by: a URL an pair of URLs a URL + a name Any of those URLs may have a query component and/or a fragment id. So there is no relationship between a particular testcase and a particular file; there are a kind of cloud of files that might be assosciated with a given test. Trying to work out what those files are is impossible in the general case; for example they can be included by document.write("some" + "resource") or similar. So it will be a huge amount of work to keep all the metadata up to date even if simple cases are handled automatically. Basically I think that if the test can work with incorrect metadata then it is highly likely that the metadata will be wrong. So I am opposed to systems that try to store lots of rarely-used metadata. I don't think that W3C needs to be in the business of regression testing browsers, so I don't see why we care about automatically deleting old results. Indeed I am skeptical about W3C publishing results at all. When W3C does publish results it should clearly be for a single revision of the browsers and a single revision of the testsuite. If newer results are avaliable the old ones could be discarded wholesale or archived somewhere well out of the way. It is the browser vendor's job to do their own day-to-day regression tracking and I don't think we need to be involved. For approved/submitted tests, I am increasingly of the opinion that using different directories is likely the wrong approach. Using two branches in the VCS combined with a commit-based review system seems like it has a number of advantages. One could still have a w3-test.org show submitted/ and approved/ directories but in order to move tests from one to the other, one would simply apply commits from the submitted branch onto the approved branch, rather than trying to copy files and run the risk of leaving stuff behind or copying the wrong bits.
Received on Tuesday, 10 May 2011 20:20:52 UTC