Re: Are self-describing reftests required?

On Aug 25, 2011, at 11:07 AM, L. David Baron wrote:

> On Thursday 2011-08-25 18:24 +0200, Øyvind Stenhaug wrote:
>> On Thu, 25 Aug 2011 17:10:57 +0200, L. David Baron
>> <dbaron@dbaron.org> wrote:
>> 
>>> I think requiring that reftests be self-describing would be a huge
>>> mistake; in many cases it significantly incresaes (i.e., multiplies
>>> by a small integer) the amount of work that the test author has to
>>> do.
>> 
>> On the other hand, having just a cryptic rendering increases the
>> amount of work any future user has to do when personally inspecting
>> individual tests, analyzing failures etc. It's a tradeoff (kind of
>> like spending effort commenting code,really). As someone who spends
>> a significant amount of time both authoring and examining testcases,
>> I certainly prefer striving to make them succinct and neat with
>> regards to both code and visual appearance.
> 
> In many cases, the easiest way to understand a reftest is to diff
> the test with the reference; the diff is often relatively
> straightforward.

In my mind the point to a self-describing reftest is that it can be evaluated either with or without the reference. It also provides a backup test that the reference is performing properly. 

Note that it's entirely fair for a reference to be a test in and of itself. If you write 20 self-describing tests that all render identically, and can't all fail the same way, then they can be references to each other with no further work.

Also, to be clear, I believe the preferred types of test are (in order): self-describing reftests, reftests, self-describing tests. In other words, not all reftests must be self-describing, but there's no reason why self-descibing tests can't also be reftests.

In general, we won't reject a new test because it's not self-describing and can only be evaluated by comparing to a reference, but there's no reason to submit a new test without a reference.

Then there's the whole issue of script tests, which are also very valuable because of their inherent automatability. We generally don't use them in CSS land because it's difficult to determine if something rendered correctly via script, but if you can use script, that would be good as well, you just have to be really sure that the script is measuring the right aspect of the test and can properly detect a failure (i.e.: just checking if a property is visible in the CSSOM tests availability and parsing, but doesn't test if the property had the intended rendering effect, however parsing tests are valuable too...).

Peter

Received on Thursday, 25 August 2011 23:39:19 UTC