Test content guidelines

Dear all,

It's good to see we now have a SVG 2 Test repository set up (thanks 
Tav!) and some concrete proposals for the document structure of the tests.

I'd like to raise the issue of the actual test content. In writing test 
cases for Gecko we've often found ourselves writing tests that, on 
success, fill the whole viewport with green (failure cases often fill it 
with red but that is less important).

This has the advantage of being really easy to determine if a test 
passed (unlike some of the tests in the SVG 1.1 test suite as I'm sure 
we're all aware, animate-elem-61-t being just one of many examples). It 
can also save resources as you can re-use the same reference file.

Unfortunately, we haven't done this very consistently in Gecko. Some 
tests fill the viewport with lime, others with green, and others just 
use a big blue rectangle. That's why I'm bringing this up now. Could we 
make it a guideline that new tests should, as far as possible, fill the 
viewport with "green" on success (i.e. rgb(0, 128, 0), "lime" is a 
little distressing)?

There are some tests where this is impractical and I definitely don't 
think we should make this a rule, just a guideline.

For those cases where it's not practical (e.g. sometimes text can be 
difficult--but even then you can get quite a way with box-shaped glyphs, 
see [1]) we might even be able to create other shared reference files to 
make inspecting test results easier.

For animated tests, in order to support user agents that don't support 
script, we could perhaps have a guideline that after 1s all animations 
should be finished/frozen and on success should fill the viewport with 
green. Scripted agents would run a script that does a seek to t=1s.

On a related note, I wonder how useful "mismatch" reference files are in 
the proposed test structure? In the above approach, anything other than 
a viewport full of green is a failure. (If there are a number of known 
reasons a test might fail, you can still make the test produce, e.g. 
different colours for each failure case to aid debugging, but I don't 
know that that really needs to be codified as separate mismatch files.)

Best regards,

Brian Birtles

[1] Example test case: 
https://bug376027.bugzilla.mozilla.org/attachment.cgi?id=593735

Received on Wednesday, 14 March 2012 01:57:35 UTC