Re: Knowing which tests are in the repository

On Fri, Aug 16, 2013 at 12:51 PM, James Graham <james@hoppipolla.co.uk> wrote:
> I already pointed out that the parser tests are examples of this. According
> to the value of the run_type query parameter they either write the input
> data using a data url, or using document.write, or using document.write one
> character at a time, or using innerHTML. Now we could of course have four
> extra files per test, each loading the test with a query string in an iframe
> (which itself runs the actual tests in further iframes). However requiring
> this kind of per-instance manual work is going to strongly discourage people
> from writing this kind of more thorough test. I can attest from experience
> that the value when implementing of these tests working in more than one way
> was immense; the document.write variants exercised codepaths that were
> untouched by the data uri-based version, and were highly effective in
> finding bugs that we might otherwise have missed (or, at least, only seen
> after doing expensive reduction of broken sites). Therefore I am totally
> against anything that would discourage this kind of extra-mile testing in
> the future.

I've written quite a lot of tests that do stuff like this.  The way I
do it is just have the test file itself do all the different variants
of testing in one run.  So when you run the parser test file, it could
just do all four test methods in sequence.  Of course, you'd need to
make sure the timeout is long enough, and perhaps allow the URL option
as a manual override in case you want to debug something.  But I don't
see why routine automated test running should care whether it runs one
long test or four shorter ones.

Received on Wednesday, 21 August 2013 12:08:21 UTC