Re: Test case review

On May 10, 2011, at 9:15 AM, Linss, Peter wrote:

> The CSS test repository contains files other than tests as well. Our test harness is not in the test repository, but we do have:
> * files referenced from tests, ie: images and stylesheets
> * reference pages (and files referenced from reference pages)
> * scripts that generate tests
> * apache server configuration files
> * the test suite build code
> 
> So yes, Shepherd will be managing and providing a review system for files other than tests as well. It will do more with a test file, like handle the test metadata, but it will have to handle other files as well.

Another thing Shepherd has to manage is the relationship between tests and their supporting files. 

For example, the way we organize our repository is that we have separate directories for tests in development, submitted for review, and approved tests. When a test is approved, it gets moved from the submitted directory to the approved directory (this allows workflows by interacting directly with the repository), when this happens, we need to make sure all the supporting files get moved into the appropriate place as well.

Another case is revision management, the harness tracks revision data and stores results against the revision of the test. When tests change, old results are automatically disregarded. The harness needs to know when a linked stylesheet, image or reference page has changed, even if the test that uses it didn't.

For the most part, we're trying to do this by getting the relationship data directly from the test (e.g. parsing the src of an <img> tag), but there are cases when the test can't refer the the necessary support file, like a fail image that only shows up if the browser is computing relative paths wrong. So we need a mechanism for declaring and tracking these relationships as well.

Received on Tuesday, 10 May 2011 17:20:23 UTC