Re: Knowing which tests are in the repository

On Thursday, August 15, 2013 at 5:27 PM, James Graham wrote:
> As far as I know we still haven't solved the problem of determining  
> which files in the repository are tests and of what type.

Yup.  
> As I  
> understand it there have been two serious proposals:
>  
> a) Use file naming conventions
> b) Use a manifest
>  
> I think b) has a number of critical advantages, which outweigh the  
> undoubted ease of authoring of a). Notably we can store extra metadata  
> which is needed to run the tests such as the comparison type for  
> reftests

That's already included within the reftests themselves.
> and unusually long timeouts for testharness tests.

Doesn't testharness already let you handle timeouts?
> Ms2ger has been implementing a variant of b) which is sufficient to work  
> out which files are tests and mark reftests with the reference type.  
> However it has not been uniformly applied across the repository. It also  
> doesn't clearly extend to extra data, or obviously lend itself to  
> automatic updates for simple cases. Therefore I propose that we try  
> something slightly different.
>  
> One option, allowing maximum code reuse, would be to base the format on  
> manifestdestiny [1]. This might also make me popular in certain circles  
> ;) However the format seems like extreme overkill; we would have to do  
> something like
>  
> [test1.html]
> type=testharness
> timeout=100
>  
> [test2.html?foo=bar]
> type=testharness
>  
> [test3.html]
> type=reftest
> reftype=equal
> ref=test3_ref.html
>  
> … and so on. A more reasonable idea might be something like
>  
> [testharness]
> test1.html timeout:100
> test2.html?foo=bar
>  
> [reftest]
> test3.html == test3_ref.html
>  
> [helpers]
> test1_1.html

A lot of the above data is redundant with existing info in the test files themselves.  
> This seems less verbose, with the exception of the helpers section. The  
> thinking behind this is that it should be possible to have a  
> manifest-update program that automagically fills in the probably-correct  
> data when a new test is added. In essence this would go through the  
> directory, look for files not in the manifest, add them as testharness  
> tests if they contain the testharness.js script, as == css tests if  
> there is a corresponding _ref file, and as helpers otherwise. Obviously  
> this isn't foolproof (it doesn't have a way to deal with manual tests,  
> for example), but does allow the easy case to be reasonably easy whilst  
> allowing fuller control in the more complex cases.
>  
> I guess one other objection is that these files will frequently cause  
> merge conflicts. I think that's a cost we will have to bear.

I think you dismiss option a) a bit too quickly.

We have ways to easily identify the different kinds of tests and reference files, the only thing we need is a way to identify non test files. It should be easy enough to agree a convention to mark those.

Manifest files are a pain to maintain and are consequently often outdated. I reckon we should try a manifest-free solution first. I'm sure we can pull it off.

--tobie

Received on Thursday, 15 August 2013 19:42:10 UTC