Re: Testing framework requirements

Hi, Garrett, Hallvord-

Thanks for the helpful comments!

Comments inline...

Hallvord R. M. Steen wrote (on 7/17/09 10:07 AM):
> On Thu, 16 Jul 2009 09:28:16 +0200, Garrett Smith
> <dhtmlkitchen@gmail.com> wrote:
>
>>> What are some of the desired requirements for an automated testing
>>> framework
>>> for testing HTML5/DOM specs from the JavaScript programmatic side?
>>
>> Script Testing Framework Features:
>> * runs in a browser

Runs in any browser, obviously.

I wouldn't preclude the idea that there could be an additional testing 
application that drives the browser and automates the testing.


>> * tests run individually, or as a suite
>> * test code and results are easy to examine
>
> I second Garrett's list - here I'd like to be even more specific:
>
> * framework should not require adding a lot of code surrounding the one
> being tested, the same test case that is used for running a test should
> be usable for debugging, setting breakpoints and stepping through

The tests should also be able to be extracted from the test suite and 
run in implementers' testing frameworks, which calls for a certain 
degree of abstraction.  I've heard many requests for this.


> This also implies my next point
> * framework should not require loading several files/frames/scripts for
> scaffolding - adds complexity to debugging (JSUnit's biggest weakness)
>
>> * suite can be paused/resumed at any time
>> * framework handles asynchronous testing
>> * events can be created and dispatched easily
>> * does not use faulty strategies (browser detection, etc).
>
> !!!

Is that a +1?


>> Suite Implementation:
>> * organized so that it is easy to find tests (searchable, sortable)
>> * search engine friendly (so that tests can be found)
>> * links to the relevant specifications, which link back to the tests

This would be very useful, and I'd love to see this happen.  We might 
need to come up with some convention for marking up the normative 
requirements in the spec itself to facilitate this, which I've started 
on for DOM3 Events.


> * It should be simple to exclude specific tests from a test run (i.e.
> useful for bugs that crash or hang the UA being tested)
> * It should be trivial to harvest result of the entire suite through a
> form POST of results in some simple format
>
> (I explicitly specify a form POST there because this is the only way to
> move such a chunk of data across domains. An obvious use case is a
> browser vendor running the tests located at www.w3.org but storing
> results from a frontend at browservendor.com/qa/testresults .)
>
> * It should give a clear and simple "score" after a test suite run,
> which will help standards compliance in general because passing the
> tests and getting a good score will be seen as important. Testing the
> browser you're using should be as simple as clicking a link, waiting a
> bit and reading the score.

This implies automated testing.  I would like to do that as much as 
possible, while keeping in mind that there are some tests that may need 
to be done manually.  The test suite should incorporate both.
Finally, it should be easy for people to create these tests, and to 
submit them for review and incorporation into the official test suite. 
Creating tests is onerous work, and it would go easier with more folks 
contributing.


>> Results of various browsers and versions could be provided on a
>> separate page.
>>
>> YUI Test is open source.
>
> ..and a pretty nice framework. I don't remember to what extent it
> depends on other YUI stuff, it would be my only gripe if it loaded
> several K of YUI code before getting to the tests. Overall, I'll by far
> prefer YUI Test to JSUnit. There are alternatives like the MochiKit test
> framework.

I'll take a closer look at YUI Test, thanks.


>> http://www.w3.org/DOM/Test/
>>
>> I do not personally find that test suite to be useful. Suggestion: Why
>> don't you post to comp.lang.javascript and ask if anyone finds the
>> test suite useful?
>
> Well, it's mostly useful to UA vendors. I can't imagine any author
> wanting to use it. However, it does tell us something that none of
> today's library authors use that framework.

W3C test suites have traditionally been aimed at implementers, as 
Hallvord says, and are meant to test the implementability of 
specifications, not conformance to those specifications, and not 
interoperability.  However, most of us involved in browser-centric 
technologies would like to change that, and recognize the need for more 
comprehensive testing.

Anyone who would like to step up and volunteer to help us move toward 
this more comprehensive testing should contact me offlist.

Regards-
-Doug Schepers
W3C Team Contact, SVG and WebApps WGs

Received on Monday, 20 July 2009 06:54:54 UTC