W3C home > Mailing lists > Public > www-dom@w3.org > July to September 2009

Re: Testing framework requirements

From: Hallvord R. M. Steen <hallvord@opera.com>
Date: Fri, 17 Jul 2009 16:07:47 +0200
To: "Garrett Smith" <dhtmlkitchen@gmail.com>, "Travis Leithead" <travil@microsoft.com>
Cc: "Doug Schepers (schepers@w3.org)" <schepers@w3.org>, "www-dom@w3.org" <www-dom@w3.org>
Message-ID: <op.uw7ti9tga3v5gv@hr-opera.oslo.opera.com>
On Thu, 16 Jul 2009 09:28:16 +0200, Garrett Smith <dhtmlkitchen@gmail.com>  

>> What are some of the desired requirements for an automated testing  
>> framework
>> for testing HTML5/DOM specs from the JavaScript programmatic side?
> Script Testing Framework Features:
> * runs in a browser
> * tests run individually, or as a suite
> * test code and results are easy to examine

I second Garrett's list - here I'd like to be even more specific:

* framework should not require adding a lot of code surrounding the one  
being tested, the same test case that is used for running a test should be  
usable for debugging, setting breakpoints and stepping through

This also implies my next point
* framework should not require loading several files/frames/scripts for  
scaffolding - adds complexity to debugging (JSUnit's biggest weakness)

> * suite can be paused/resumed at any time
> * framework handles asynchronous testing
> * events can be created and dispatched easily
> * does not use faulty strategies (browser detection, etc).


> Suite Implementation:
>  * organized so that it is easy to find tests (searchable, sortable)
>  * search engine friendly (so that tests can be found)
>  * links to the relevant specifications, which link back to the tests

* It should be simple to exclude specific tests from a test run (i.e.  
useful for bugs that crash or hang the UA being tested)
* It should be trivial to harvest result of the entire suite through a  
form POST of results in some simple format

(I explicitly specify a form POST there because this is the only way to  
move such a chunk of data across domains. An obvious use case is a browser  
vendor running the tests located at www.w3.org but storing results from a  
frontend at browservendor.com/qa/testresults .)

* It should give a clear and simple "score" after a test suite run, which  
will help standards compliance in general because passing the tests and  
getting a good score will be seen as important. Testing the browser you're  
using should be as simple as clicking a link, waiting a bit and reading  
the score.

> Results of various browsers and versions could be provided on a separate  
> page.
> YUI Test is open source.

..and a pretty nice framework. I don't remember to what extent it depends  
on other YUI stuff, it would be my only gripe if it loaded several K of  
YUI code before getting to the tests. Overall, I'll by far prefer YUI Test  
to JSUnit. There are alternatives like the MochiKit test framework.

> http://www.w3.org/DOM/Test/
> I do not personally find that test suite to be useful. Suggestion: Why
> don't you post to comp.lang.javascript and ask if anyone finds the
> test suite useful?

Well, it's mostly useful to UA vendors. I can't imagine any author wanting  
to use it. However, it does tell us something that none of today's library  
authors use that framework.

Hallvord R. M. Steen, Core Tester, Opera Software
http://www.opera.com http://my.opera.com/hallvors/
Received on Friday, 17 July 2009 14:07:46 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 20 October 2015 10:46:14 UTC