Re: Do test titles really need to be unique?

On Jul 30, 2013, at 2:20 PM, Rebecca Hauck wrote:

> Hi all,
> (but this is probably mostly for Peter, maybe Elika)
> 
> I'm reviewing and updating some of our testing documentation that is eventually going to be centralized in one location used for all W3C testing, not just CSS testing.  Do we really need to enforce unique titles on tests?  I don't believe there is any such requirement for non-CSS tests and I assume Shepherd doesn't have al problem with duplicate titles since each test file is seen as a separate asset. 
> 
> If we can soften this requirement, it's one less thing authors and reviewers need to deal with.   
> 
> Thoughts?

So the answer is yes and no. It used to be a hard requirement and Shepherd considered any two files with the same name as being the same test. I recently relaxed that restriction in Shepherd and it now examines the contents of files that have the same name to see if they're really the same thing or not.

The build code still enforces the unique name within a given test suite. This is important because all the tests get put into one directory and if there are name collisions one of them would get overridden. The unique name across the entire repository was to allow us to build suites that combine tests from multiple specs (to test a profile, for example) so name collisions between suites was important too.

The harness also currently requires all test names to be unique, even in different suites, as it tracks the results per test name so that results can be shared between test suites that use the same test file.

The plan for the new build code (that's in progress) is to relax the file name restriction and just deal with collisions at build time (probably by putting the files in different directories). I also plan to change the harness to track results by file content hashes rather than test names, so the restriction can be relaxed there too.

So the bottom line is, it still matters for parts of the infrastructure, but so long as the name collisions are in different suites it's not a problem (but could be in the future if we combine suites before all the changes are in place to support that). But the intention is to remove the restriction.

It is still however, a good practice to use unique names as it causes a lot less confusion when people refer to tests.

We also have a whole lot of recent submissions that don't adhere to our naming conventions. I think those should be fixed.

Peter