Re: Test content guidelines

Hi Brian,

On Wed, Mar 14, 2012 at 12:57 PM, Brian Birtles <bbirtles@mozilla.com>wrote:

> Dear all,
>
> It's good to see we now have a SVG 2 Test repository set up (thanks Tav!)
> and some concrete proposals for the document structure of the tests.
>

+1 - thanks Tav!


> I'd like to raise the issue of the actual test content. In writing test
> cases for Gecko we've often found ourselves writing tests that, on success,
> fill the whole viewport with green (failure cases often fill it with red
> but that is less important).
>

This is a good idea. At least consistency in pass/fail colour indication
would be great.


> This has the advantage of being really easy to determine if a test passed
> (unlike some of the tests in the SVG 1.1 test suite as I'm sure we're all
> aware, animate-elem-61-t being just one of many examples). It can also save
> resources as you can re-use the same reference file.
>
> Well, some reference files. I'd expect test output to contain a variation
in graphics for many cases...


> Unfortunately, we haven't done this very consistently in Gecko. Some tests
> fill the viewport with lime, others with green, and others just use a big
> blue rectangle. That's why I'm bringing this up now. Could we make it a
> guideline that new tests should, as far as possible, fill the viewport with
> "green" on success (i.e. rgb(0, 128, 0), "lime" is a little distressing)?
>
> There are some tests where this is impractical and I definitely don't
> think we should make this a rule, just a guideline.
>
> For those cases where it's not practical (e.g. sometimes text can be
> difficult--but even then you can get quite a way with box-shaped glyphs,
> see [1]) we might even be able to create other shared reference files to
> make inspecting test results easier.
>

SVG Tiny fonts would ease the pain here:-)


> For animated tests, in order to support user agents that don't support
> script, we could perhaps have a guideline that after 1s all animations
> should be finished/frozen and on success should fill the viewport with
> green. Scripted agents would run a script that does a seek to t=1s.
>

I think 1s may be too quick. It's OK for automated tests, but people may
actually want to watch the animation and 1s doesn't seem to be very long
especially
for complex/chained animations, etc.

But in principle it makes sense.


> On a related note, I wonder how useful "mismatch" reference files are in
> the proposed test structure? In the above approach, anything other than a
> viewport full of green is a failure. (If there are a number of known
> reasons a test might fail, you can still make the test produce, e.g.
> different colours for each failure case to aid debugging, but I don't know
> that that really needs to be codified as separate mismatch files.)
>
> Best regards,
>
> Brian Birtles
>
> [1] Example test case: https://bug376027.bugzilla.**
> mozilla.org/attachment.cgi?id=**593735<https://bug376027.bugzilla.mozilla.org/attachment.cgi?id=593735>


That's html with embedded SVG, poor SVG only UAs!

Alex