Re: Testing framework requirements

On Thu, 16 Jul 2009 09:28:16 +0200, Garrett Smith <dhtmlkitchen@gmail.com>  
wrote:

>> What are some of the desired requirements for an automated testing  
>> framework
>> for testing HTML5/DOM specs from the JavaScript programmatic side?
>
> Script Testing Framework Features:
> * runs in a browser
> * tests run individually, or as a suite
> * test code and results are easy to examine

I second Garrett's list - here I'd like to be even more specific:

* framework should not require adding a lot of code surrounding the one  
being tested, the same test case that is used for running a test should be  
usable for debugging, setting breakpoints and stepping through

This also implies my next point
* framework should not require loading several files/frames/scripts for  
scaffolding - adds complexity to debugging (JSUnit's biggest weakness)

> * suite can be paused/resumed at any time
> * framework handles asynchronous testing
> * events can be created and dispatched easily
> * does not use faulty strategies (browser detection, etc).

!!!

> Suite Implementation:
>  * organized so that it is easy to find tests (searchable, sortable)
>  * search engine friendly (so that tests can be found)
>  * links to the relevant specifications, which link back to the tests


* It should be simple to exclude specific tests from a test run (i.e.  
useful for bugs that crash or hang the UA being tested)
* It should be trivial to harvest result of the entire suite through a  
form POST of results in some simple format

(I explicitly specify a form POST there because this is the only way to  
move such a chunk of data across domains. An obvious use case is a browser  
vendor running the tests located at www.w3.org but storing results from a  
frontend at browservendor.com/qa/testresults .)

* It should give a clear and simple "score" after a test suite run, which  
will help standards compliance in general because passing the tests and  
getting a good score will be seen as important. Testing the browser you're  
using should be as simple as clicking a link, waiting a bit and reading  
the score.


> Results of various browsers and versions could be provided on a separate  
> page.
>
> YUI Test is open source.

..and a pretty nice framework. I don't remember to what extent it depends  
on other YUI stuff, it would be my only gripe if it loaded several K of  
YUI code before getting to the tests. Overall, I'll by far prefer YUI Test  
to JSUnit. There are alternatives like the MochiKit test framework.

> http://www.w3.org/DOM/Test/
>
> I do not personally find that test suite to be useful. Suggestion: Why
> don't you post to comp.lang.javascript and ask if anyone finds the
> test suite useful?

Well, it's mostly useful to UA vendors. I can't imagine any author wanting  
to use it. However, it does tell us something that none of today's library  
authors use that framework.

-- 
Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/

Received on Friday, 17 July 2009 14:07:46 UTC