W3C home > Mailing lists > Public > www-dom@w3.org > July to September 2009

Re: Testing framework requirements

From: Hallvord R. M. Steen <hallvord@opera.com>
Date: Fri, 17 Jul 2009 16:07:47 +0200
To: "Garrett Smith" <dhtmlkitchen@gmail.com>, "Travis Leithead" <travil@microsoft.com>
Cc: "Doug Schepers (schepers@w3.org)" <schepers@w3.org>, "www-dom@w3.org" <www-dom@w3.org>
Message-ID: <op.uw7ti9tga3v5gv@hr-opera.oslo.opera.com>
On Thu, 16 Jul 2009 09:28:16 +0200, Garrett Smith <dhtmlkitchen@gmail.com>  
wrote:

>> What are some of the desired requirements for an automated testing  
>> framework
>> for testing HTML5/DOM specs from the JavaScript programmatic side?
>
> Script Testing Framework Features:
> * runs in a browser
> * tests run individually, or as a suite
> * test code and results are easy to examine

I second Garrett's list - here I'd like to be even more specific:

* framework should not require adding a lot of code surrounding the one  
being tested, the same test case that is used for running a test should be  
usable for debugging, setting breakpoints and stepping through

This also implies my next point
* framework should not require loading several files/frames/scripts for  
scaffolding - adds complexity to debugging (JSUnit's biggest weakness)

> * suite can be paused/resumed at any time
> * framework handles asynchronous testing
> * events can be created and dispatched easily
> * does not use faulty strategies (browser detection, etc).

!!!

> Suite Implementation:
>  * organized so that it is easy to find tests (searchable, sortable)
>  * search engine friendly (so that tests can be found)
>  * links to the relevant specifications, which link back to the tests


* It should be simple to exclude specific tests from a test run (i.e.  
useful for bugs that crash or hang the UA being tested)
* It should be trivial to harvest result of the entire suite through a  
form POST of results in some simple format

(I explicitly specify a form POST there because this is the only way to  
move such a chunk of data across domains. An obvious use case is a browser  
vendor running the tests located at www.w3.org but storing results from a  
frontend at browservendor.com/qa/testresults .)

* It should give a clear and simple "score" after a test suite run, which  
will help standards compliance in general because passing the tests and  
getting a good score will be seen as important. Testing the browser you're  
using should be as simple as clicking a link, waiting a bit and reading  
the score.


> Results of various browsers and versions could be provided on a separate  
> page.
>
> YUI Test is open source.

..and a pretty nice framework. I don't remember to what extent it depends  
on other YUI stuff, it would be my only gripe if it loaded several K of  
YUI code before getting to the tests. Overall, I'll by far prefer YUI Test  
to JSUnit. There are alternatives like the MochiKit test framework.

> http://www.w3.org/DOM/Test/
>
> I do not personally find that test suite to be useful. Suggestion: Why
> don't you post to comp.lang.javascript and ask if anyone finds the
> test suite useful?

Well, it's mostly useful to UA vendors. I can't imagine any author wanting  
to use it. However, it does tell us something that none of today's library  
authors use that framework.

-- 
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
Received on Friday, 17 July 2009 14:07:46 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Friday, 22 June 2012 06:14:02 GMT