Knowing which tests are in the repository

As far as I know we still haven't solved the problem of determining 
which files in the repository are tests and of what type. As I 
understand it there have been two serious proposals:

a) Use file naming conventions
b) Use a manifest

I think b) has a number of critical advantages, which outweigh the 
undoubted ease of authoring of a). Notably we can store extra metadata 
which is needed to run the tests such as the comparison type for 
reftests and unusually long timeouts for testharness tests.

Ms2ger has been implementing a variant of b) which is sufficient to work 
out which files are tests and mark reftests with the reference type. 
However it has not been uniformly applied across the repository. It also 
doesn't clearly extend to extra data, or obviously lend itself to 
automatic updates for simple cases. Therefore I propose that we try 
something slightly different.

One option, allowing maximum code reuse, would be to base the format on 
manifestdestiny [1]. This might also make me popular in certain circles 
;) However the format seems like extreme overkill; we would have to do 
something like

[test1.html]
type=testharness
timeout=100

[test2.html?foo=bar]
type=testharness

[test3.html]
type=reftest
reftype=equal
ref=test3_ref.html

… and so on. A more reasonable idea might be something like

[testharness]
test1.html timeout:100
test2.html?foo=bar

[reftest]
test3.html == test3_ref.html

[helpers]
test1_1.html

This seems less verbose, with the exception of the helpers section. The 
thinking behind this is that it should be possible to have a 
manifest-update program that automagically fills in the probably-correct 
data when a new test is added. In essence this would go through the 
directory, look for files not in the manifest, add them as testharness 
tests if they contain the testharness.js script, as == css tests if 
there is a corresponding _ref file, and as helpers otherwise. Obviously 
this isn't foolproof (it doesn't have a way to deal with manual tests, 
for example), but does allow the easy case to be reasonably easy whilst 
allowing fuller control in the more complex cases.

I guess one other objection is that these files will frequently cause 
merge conflicts. I think that's a cost we will have to bear.

[1] https://mozbase.readthedocs.org/en/latest/manifestdestiny.html

Received on Thursday, 15 August 2013 15:28:22 UTC