Test content guidelines (was: Minutes, 20 April SVG telcon)

Sorry, I forgot to change the subject after re-sending. Please reply 
under this message.

(2012/04/20 11:27), Brian Birtles wrote:
> Hi all,
>
> Thanks for discussing the test content guidelines subject I brought up.
>
> I'll follow-up inline with a few comments.
>
> (2012/04/20 7:40), Chris Lilley wrote:
>  > test content guidelines
>  >
>  > heycam: brian was aksing for guidelines so they can succeed or
>  > fail in the same way like all green rectangles
>  >
>  > tav: like red and green but if they all look the same it is
>  > confusing
>  > ... doesnt say what is being tested
>  > ... dont want a rect that covers the whole thing
>  >
>  > heycam: if the tests are automated you won't need to look at
>  > them
>  >
>  > Tav: so there is no debugging help
>
> For most tests, I've found it's not needed. Generally you should focus
> your tests so they test one thing.
>
> For cases where there are a number of possible failure scenarios you can
> have tests which flood the viewport with green on success, purple on
> failure scenario 1, orange on failure scenario 2 etc.
>
> That still gives you the advantages of:
>
> (1) Easier manual inspection of individual tests (the current test suite
> is particularly weak in this regard since the success condition differs
> from test to test and is often very complex)
> (2) Easier generation of reference images (a no-op)
> (3) No edges that give pixel differences due to anti-aliasing
> (4) Quicker feedback when running automated suites (as soon as you see
> something other than green on the screen you know you've got issues)
> (5) Fewer resources required (in terms of number of files, number of
> renders required etc.)
> (6) Easier to write tests (less inventiveness required)
>
>  > Cyril: point is to automate error detection
>
> It helps with this, but I think the advantages for manual inspection of
> test results are greater.
>
>  > heycam: for animation, brian suggests tests where the final
>  > state is at one second so there is a snapshot to compare to the
>  > ref
>  > ... script can sett the time to that point
>
> I think it would be good to standardise the snapshot time where
> possible. There will, of course, be cases where we deviate, but having a
> standard time makes understanding the tests and debugging simpler if you
> know that generally, for example, t=5s is the key moment.
>
>  > ChrisL: for path animations you need multiple snapshots surely
>
> For most animation reftests I've found one snapshot is sufficient. For
> cases where you actually want to test values over time I've found it
> more efficient to use a purely scripted test where you repeatedly seek
> the timeline and query the values you're interested in. Generally,
> there's no need to render the whole scene multiple times since it's just
> one or two values that you care about.
>
> For non-scripted UAs if you design the tests to test one specific thing
> each (rather than just a series of samples of an animation that are
> effectively testing the same things) then I think the number of cases
> where you actually want multiple snapshots of the same animation will be
> small. I think. :)
>
>  > tav: examples of animation reftests?
>
> There are hundreds here:
> http://mxr.mozilla.org/mozilla-central/source/layout/reftests/svg/smil/
>
> They all use script to set the snapshot time. It would be good to just
> declare that in markup so non-scripted UAs can run the tests.
>
>  > heycam: ok so aiming for a single green or red rect is not
>  > good, but if it is a simple pass/fail result then go for the
>  > rect
>
> I'm less concerned about the red flood fill. For transforms, for
> example, you could fill the canvas with red, then transform a green rect
> so that it should fill the canvas.
>
> There will *definitely* be some cases where you don't want the green
> flood fill, but, depending on the section of the spec, I think there
> will not be too many. Have a look at:
>
> http://mxr.mozilla.org/mozilla-central/source/layout/reftests/svg/smil/reftest.list
>
>
> On the right hand side you see "anim-standard-ref.svg" and "lime.svg"
> over and over again. Go into the subfolders and it's the same, e.g. the
> 'syncbase' folder uses "green-box-ref.svg/xhtml" as the reference image
> for all 85 tests.
>
> The fact that there are multiple standard reference images,
> "anim-standard-ref.svg", "lime.svg", "green-box-ref.svg" etc. is
> something I want to avoid in the SVG test suite and one of the reasons I
> brought this topic up. We should just have green.svg.
>
>  > ChrisL: sometimes you can loose track if all the tests look the
>  > same
>  >
>  > (general agreement)
>
> I don't understand this. Can someone explain?
>
> I expect the SVG 2 test suite will contain thousands and thousands of
> test files. In that case, you keep track of them by name, not
> appearance. But perhaps I've missed the point here.
>
>
> I don't want a hard rule about "you must have success = green flood
> fill" or anything like that. I'm just trying to avoid:
>
> (1) The current SVG test suites where it's really hard to tell at a
> glance if a test has passed.
> (2) The situation we got into with Gecko where we have a number of
> "standard" reference files.
>
>
> Regards,
>
> Brian Birtles
>

Received on Friday, 20 April 2012 02:31:20 UTC