Re: test writing guidelines regarding use of red

On Wed, 20 Oct 2010 05:01:35 +0200, Cameron McCormack <cam@mcc.id.au>  
wrote:

> Hi.
>
> I’m reviewing some of the Microsoft test submissions.  Take
> styling-pres-05-f as an example.  The test is good in that red is used
> to indicate test failure, but when the test succeeds the shapes (that
> would otherwise be red if the test failed) are set to fill:none.  The
> resulting test slide if the test passes is blank apart from the CVS
> revision.  Do we have test writing guidelines written down somewhere?
> I’m thinking we should be following rules like:
>
>   * If any red is showing, the test failed.
>   * If any green is showing and no red is showing, the test passed.
>   * Otherwise, you need to read the pass criteria.

I agree with these principles, that's what I've been trying to follow so  
far. Here are a few more suggestions:

   * Make the test as targetted to a particular assertion as possible,  
without involving other functionality if it can be avoided (basically try  
to keep it simple, the recently submitted Microsoft tests are pretty good  
examples of this)
   * Don't make tests that take long to run (if they take long to run  
consider splitting the test into several tests instead)
   * Don't add text saying FAIL in green, or things like "This must be red"  
as a pass criteria
   * Indicate prerequisites, e.g fonts, scripting support etc. (Thought:  
maybe we could use @font-face to reference the fonts for some of the tests  
that currently require installing fonts on the system.)
   * If the passcriteria can usefully be checked via scripting then we  
should do that (as well as providing some visual feedback), and in  
particular we should try to use the new testing framework provided by  
test.w3.org.

I've put some initial documentation for how to use the new testing  
framework on the svg wiki[1]. A summary:

- Add a script element to include the testing framework
- Use the testing framework to report each assertion in the test
- I'd suggest that the tests should still show green for pass and red for  
fail, even if the test status is reported to the testing framework

The documentation for how to write tests using the framework can be found  
at the top of testharness.js[2]. So far there are two examples in the  
testsuite that attempts to use the framework:

- http://dev.w3.org/SVG/profiles/1.1F2/test/svg/struct-dom-01-b.svg
    Diff:  
http://dev.w3.org/cvsweb/SVG/profiles/1.1F2/test/svg/struct-dom-01-b.svg.diff?r1=1.5&r2=1.6&f=h
- http://dev.w3.org/SVG/profiles/1.1F2/test/svg/struct-svg-02-f.svg (note:  
still a work-in-progress)
    Diff:  
http://dev.w3.org/cvsweb/SVG/profiles/1.1F2/test/svg/struct-svg-02-f.svg.diff?r1=1.1&r2=1.2&f=h

Cheers
/Erik

[1] http://www.w3.org/Graphics/SVG/WG/wiki/Testing_Framework
[2] http://dev.w3.org/SVG/profiles/1.1F2/test/resources/testharness.js

-- 
Erik Dahlstrom, Core Technology Developer, Opera Software
Co-Chair, W3C SVG Working Group
Personal blog: http://my.opera.com/macdev_ed

Received on Thursday, 21 October 2010 08:36:50 UTC