Re: WebTV Help for Getting Engaged in W3C Test Effort

On 01/05/2014 23:20, James Graham wrote:
> If we wanted globally unique ids, it would probably mean using 
> something like a random uuid. For example we could give each test a 
> name like aec2cf60-d17a-11e3-80c1-cbadd29e6cd4. If we did that for 
> both filenames and the names of tests within files we would have a way 
> of identifying every test that wasn't prone to collisions. This is 
> quite similar to the way that git and other version control systems 
> work under the hood. I hope it's obvious that this setup would be 
> awful in practice; people would resent the overhead of generating 
> these names and refuse to submit tests to the testsuite, 
As you say, using GUIDs and means of generating them is a well 
understood problem. The style and length of the GUID, the means to 
generate it, and the probability of a clash would all need discussing 
but this could be done. That said, I lean towards the "allow clashes and 
resolve downstream" approach - see below.

> any attempt to communicate using the ids would be a nightmare,
Are you referring to the fact that the ID is not human meaningful or 
that it's long? I'm not sure either issue is a problem. We're all used 
to identifying resources using long opaque strings (e.g. a Google doc 
URL), referring to bugs in issue trackers using meaningless integers (or 
whatever the tracker uses), referring to projects using code names, 
referring to customer orders using Sales Force IDs, .... the examples 
are countless. The abstraction of the stable identifier from human 
meaningful description/title is *useful* as it enforces precision.

> and people who did bother to submit tests would likely copy ids 
> between files rather than generating new ones each time. 
There's a whole bunch of things you're asking people to do correctly 
when submitting a test case - understand good test design, use the test 
harness properly, read the spec thoroughly and then actually cut the 
code, review the code... This typically takes many minutes or hours. Are 
the few seconds of creating a test ID going to put people off or be done 
any more incorrectly than the other things?

> For files that generate multiple tests from data it would be even 
> worse; it's not clear at all how to generate a unique, but stable, id 
> for each test in that case.
I accept that point and I'd need to understand the details better to 
meaningfully comment. The general problem is solvable - this is done in 
the HbbTV case using a templating mechanism - but might be harder for 
the generated W3C tests.

>
> The other option is to allow the possibility of name clashes, but make 
> them easy to resolve. One way to do this would be to require one test 
> per file and use the path to the file as the unique test identifier. 
> It is possible that two people could simultaneously submit different 
> tests with the same name, but if it happened the conflict would be 
> rather easy to resolve. However this system has one showstopper-level 
> disadvantage; the lack of multiple tests per file makes test authors 
> dramatically less productive, so we lose significant coverage. That's 
> not an acceptable tradeoff.
Agreed - I can see that multiple tests per a file is useful and not 
something that would make sense to change.

The next logical step (to me) is IDs attributed to tests, with the 
filename playing no role in the ID. Uniqueness of this ID can be checked 
at commit and/or build time. The test name would still be retained as it 
is today with all the advantages you lay out below.

The way HbbTV approaches ID creation is to namespace the ID, with each 
contributing company/person given a unique prefix and then within that 
name-space it is up to the contributor to coordinate amongst themselves 
when creating IDs. There are other ways, but the key point is that 
creating and using opaque and unique IDs needn't be complicated or time 
consuming.

>
> So finally we come to the solution where we allow multiple tests per 
> file, and give each test a human-readable name. This only requires 
> local coordination (we need to ensure that each file name is unique 
> and each test name is unique, but don't need to compare to any global 
> state), doesn't require using human-unfriendly identifiers like uuids, 
> and allows test authors to be productive by supporting many tests in a 
> file. It also has some more side benefits; by requiring that each test 
> has a unique title we get some metadata about what is being tested for 
> "free". This dramatically reduces the need to have a separate 
> description of the test intent. Clearly this solution is a tradeoff, 
> but it's one that works well.

Received on Friday, 2 May 2014 09:45:09 UTC