Re: Reftest review in test harness

On Apr 28, 2012, at 11:55 PM, Aryeh Gregor wrote:

> Currently, the test harness handles reftests by having two tabs that
> you can manually flip between, one of which contains the test file and
> one of which contains the reference file.  This is not an ideal way to
> manually review reftests.  First of all, it's slow; second of all, it
> makes it hard to spot subtle errors.  A better way to manually review
> reftests is for the images to flip back and forth automatically.  It
> fails if there's any visible flickering, and passes if there isn't.
> This way, it's possible to reliably check even multiple tests per
> second.  A crude implementation I wrote for my transforms reftests is
> here:
> 
> http://hg.csswg.org/test/raw-file/77afcb7689ae/contributors/aryehgregor/incoming/viewer.html
> 
> Is there any interest in updating the CSSWG test harness to have
> similar functionality?
> 

That's already on the list of improvements for the harness. Along with automatic collection of results from script tests using testharness.js.

The harness was originally designed under some fairly tight restrictions. Because its purpose is to test various functionality in browsers, it has to minimize the functionality of browsers that it requires to operate. Most notably, it has to work with script turned off. (Remember, it has to work in just about anything capable of rendering web content, not just the current versions of the four major browsers.)

That said, it's perfectly fair to "discover" additional capabilities of the browser at run time and augment the behavior accordingly. The plan is to keep the existing reference tabs, but when scripting is available, also present a "flicker" option that can be turned on and off.

At the moment, I'm up to my elbows in the bowels of Shepherd. I just finished improving the way cross-asset file renames are handled, and now I'm reworking the way it handles merge contexts (I found some issues there). Once I land all that I can switch back to the harness for a while. There's a bunch of code written for the W3C fork of the harness that I want to backport and I also want to bring the harness front-end code more inline with core code improvements I made while working on Shepherd. Adding the reftest flicker shouldn't be all that much work. There are also a bunch of improvements to the test suite build code on deck as well as lots more Shepherd functionality...

Peter

Received on Sunday, 29 April 2012 17:55:18 UTC