RE: HTML Testing Task Force Conf Call Agenda 9/7/2010

A ref test will work in some simple cases, though it will not work in all cases.
For example you can't build a ref test for an audio or video test.

Ref tests also fail to catch bugs when you have a regression in a lower parts of the rendering stack.
In your example if you fail to render text, e.g. page is blank.
The test will still pass because both the test and ref test will not have any text displayed.

So technically if you don't render anything you can always pass all the ref tests.
As such I'd rather have tests be self-describing as James states below.

-Kris

-----Original Message-----
From: James Graham [mailto:jgraham@opera.com] 
Sent: Monday, September 13, 2010 2:01 AM
To: Ms2ger
Cc: Kris Krueger; 'public-html-testsuite@w3.org'
Subject: Re: HTML Testing Task Force Conf Call Agenda 9/7/2010

On 09/12/2010 09:24 PM, Ms2ger wrote:
> Kris Krueger wrote:
>> <krisk> since you need many 'references' - one for a unix box, one 
>> for a phone, one for MAC, etc...
>> <krisk> The ref image ends up not really being a reference at all
>
> I think you are misunderstanding how reftests work. For example, if 
> you have
>
> test.html
> [[[
> <p>There should be one line of text below this one.
> <p>This line should be visible
> <p hidden>This line should not be visible ]]]
>
> test.ref.html
> [[[
> <p>There should be one line of text below this one.
> <p>This line should be visible
> ]]]
>
> that would be sufficient for all platforms. Indeed, Mozilla runs a 
> test along these lines on every checkin, on Windows, Mac and Linux, 
> and on mobile phones.
>

Indeed. Furthermore the reftest should be self-describing (not all Mozilla reftests meet this criterion but the CSS WG requires it for reftests that form part of their official testsuite and I believe we should do the same). This means that you can always run the tests as if they were pure visual tests simply by reading the description. This eliminates any residual cross-platform concerns that might exist.

Received on Monday, 13 September 2010 23:44:41 UTC