Re: Experimental test harness

On Thu, 08 Mar 2007 17:40:35 +0100, Dominique Hazael-Massieux <dom@w3.org>  
wrote:

>
> Hi,
>
> Following-up on our discussions last week on setting up an experimental
> test harness that would allow to navigate through test cases and record
> results, I've set up such a script at:
> http://www.w3.org/2007/03/mth/harness
>
> At this time, only the CSS MP link is functional - the DOM one can't be
> used due to the way the Javascript displays the result.
>
> The harness drives the user through the set of known test cases,
> recording at each step which test case passes and which doesn't, leading
> to a results table such as:
> http://www.w3.org/2007/03/mth/results?ts=cssmp
>
> Of course, this is still very drafty and could use quite a few
> improvements; a few of the ideas that come to my mind:
>  * allow to define more context for individual test cases; at this time,
> the context for each test case is very crudely defined; I would need to
> define what headers the content should be sent with (e.g. content type)
>  * attach more metadata to the list of testcases to make the results
> table more interesting
>  * bind the data to abstract user agents rather than to a unique user
> agent string (I'm thinking that WURFL should be able to help for this)
>  * allow a given user to start from a given test case, giving hints on
> which test cases haven't been run on his/her device, or which test cases
> have received inconsistent results
>  * allow to skip a test case (i.e. going from test n to test n+2) when a
> given test can't be run
>
> Feedback and suggestions welcome; please do keep in mind this is
> entirely experimental at this stage.

Nice work, Dom!

A couple of quick thoughts:

Would it be possible to choose DOCTYPE to be XHTML, HTML, or none?
I haven't understood the result collecting, but the current harness sends  
the result to the next test?
It would be nice to have an overview of the testing progress. Like "Test  
5/77" at the bottom.
A link to results page would be helpful (e.g., a button).
A session ID (or rather user) would enable testing over several days.

Till

>
> Dom
>
>



-- 
Till Halbach
Quality Assurance, Opera Software (www.opera.com)

Received on Tuesday, 20 March 2007 15:59:12 UTC