Re: HTML Testing Task Force Conf Call Agenda 9/7/2010

On 09/14/2010 01:43 AM, Kris Krueger wrote:
> A ref test will work in some simple cases, though it will not work in all cases.
> For example you can't build a ref test for an audio or video test.

For an audio test, sure. For a video test you can get a surprisingly 
long way e.g. set the test up to run the video for a certain amount of 
time and check the rendering at that time against a reference. It is 
slightly more complex but I don't think it is impossible.

> Ref tests also fail to catch bugs when you have a regression in a lower parts of the rendering stack.
> In your example if you fail to render text, e.g. page is blank.
> The test will still pass because both the test and ref test will not have any text displayed.
>
> So technically if you don't render anything you can always pass all the ref tests.
> As such I'd rather have tests be self-describing as James states below.

Right, there are two complementary solutions to this. One is to make the 
tests self-describing as I already stated. The second is to make 
ordinary "visual" tests for the reference images. Then if the reference 
fails, you have to manually inspect all the tests referring to it, 
rather than running them automatically. Assuming you take pains to reuse 
the same reference images as far as possible you can have large numbers 
of reftests keying off a tiny number of visual tests. You win big in the 
typical case (the reference, which is simple by definition, doesn't 
fail) and suffer no penalty compared to not using reftests in the worst 
case.

The idea with reftests is not to eliminate all tests that require human 
interaction, but to dramatically reduce their number in typical cases.

Received on Tuesday, 14 September 2010 09:12:43 UTC