Manual test cleanup [was: Re: Knowing which tests are in the repository]

On 27/08/13 13:35, James Graham wrote:
> On 23/08/13 14:16, Robin Berjon wrote:
>> Hi James,
>>
>> On 22/08/2013 18:31 , James Graham wrote:
>>> A modified proposal:
>>
>> Overall, I like this. It seems to address issues with maintainability
>> and developer-friendliness well. Just a few minor quibbles below:
>>
>>> * Any html, xhtml or svg file that links to testharness.js is a
>>> testharness test
>>
>> I wonder if we can brutally simplify this to be for (X)HTML and SVG
>> files that contain the string "testharness.js". It avoids parsing and is
>> likely correct.
>
> I wonder if the cost of parsing is still a problem if you assume that
> instead of rewriting the whole manifest each time, you store the SHA1 of
> the previous update and only look at files that have changed since then
> (using git diff --name-only {PREVIOUS_REF} HEAD). The initial import
> will be slow, but subsequent updates will generally be fast.

I have now written a tool that implements much of this. It will detect 
reftests and testharness tests based on the file contents as well as 
manual tests based on the name. It also supports incremental updates, 
but does not yet have the ability to override specific data.

The work in progress (warning: still buggy) is at [1]. Some example 
output is at [2].

The problem is that at the moment much of the data, particularly 
relating to manual tests, is incorrect. It would be helpful if you have 
ever submitted a manual test to the repository to take a moment and 
change the name so it has the -manual suffix. This is particularly 
important if the test includes testharness.js (although such cases are 
also relatively easy to detect since they will cause timeouts).

This convention also needs to be added to the documentation.

[1] 
https://github.com/jgraham/html-testsuite/blob/manifest/tools/scripts/manifest.py

[2] https://gist.github.com/jgraham/6440204

Received on Wednesday, 4 September 2013 17:39:02 UTC