Re: Knowing which tests are in the repository

On 21/08/13 13:07, Aryeh Gregor wrote:
> On Fri, Aug 16, 2013 at 12:51 PM, James Graham <james@hoppipolla.co.uk> wrote:
>> I already pointed out that the parser tests are examples of this. According
>> to the value of the run_type query parameter they either write the input
>> data using a data url, or using document.write, or using document.write one
>> character at a time, or using innerHTML. Now we could of course have four
>> extra files per test, each loading the test with a query string in an iframe
>> (which itself runs the actual tests in further iframes). However requiring
>> this kind of per-instance manual work is going to strongly discourage people
>> from writing this kind of more thorough test. I can attest from experience
>> that the value when implementing of these tests working in more than one way
>> was immense; the document.write variants exercised codepaths that were
>> untouched by the data uri-based version, and were highly effective in
>> finding bugs that we might otherwise have missed (or, at least, only seen
>> after doing expensive reduction of broken sites). Therefore I am totally
>> against anything that would discourage this kind of extra-mile testing in
>> the future.
>
> I've written quite a lot of tests that do stuff like this.  The way I
> do it is just have the test file itself do all the different variants
> of testing in one run.  So when you run the parser test file, it could
> just do all four test methods in sequence.  Of course, you'd need to
> make sure the timeout is long enough, and perhaps allow the URL option
> as a manual override in case you want to debug something.  But I don't
> see why routine automated test running should care whether it runs one
> long test or four shorter ones.
>

Yes, that does work, but in practice your tests have been problematic on 
automated infrastructure due to the long run time; the DOM Range tests 
are an example of this, I believe. It would be preferable to be able to 
specify that they should run in chunks without either forcing you to 
break them into multiple files, or to reduce coverage, or to have 
additional project-specific annotations chunking the tests.

Received on Wednesday, 21 August 2013 12:15:33 UTC