W3C home > Mailing lists > Public > www-dom@w3.org > July to September 2009

Re: Testing framework requirements

From: Garrett Smith <dhtmlkitchen@gmail.com>
Date: Mon, 20 Jul 2009 12:47:29 -0700
Message-ID: <c9e12660907201247p67d522f5w7f087027d9001c31@mail.gmail.com>
To: Doug Schepers <schepers@w3.org>
Cc: "Hallvord R. M. Steen" <hallvord@opera.com>, Travis Leithead <travil@microsoft.com>, "www-dom@w3.org" <www-dom@w3.org>
On Sun, Jul 19, 2009 at 11:54 PM, Doug Schepers<schepers@w3.org> wrote:
> Hi, Garrett, Hallvord-
> Thanks for the helpful comments!
> Comments inline...
> Hallvord R. M. Steen wrote (on 7/17/09 10:07 AM):
>> On Thu, 16 Jul 2009 09:28:16 +0200, Garrett Smith
>> <dhtmlkitchen@gmail.com> wrote:
>>>> What are some of the desired requirements for an automated testing
>>>> framework
>>>> for testing HTML5/DOM specs from the JavaScript programmatic side?
>>> Script Testing Framework Features:
>>> * runs in a browser
> Runs in any browser, obviously.

How would you run a javascript-driven test runner in a browser that
does not have an implementation of ECMAScript?

I would guess some browsers, such as Blackberry or Opera Mobile, may
present some challenges for an automation test.

> I wouldn't preclude the idea that there could be an additional testing
> application that drives the browser and automates the testing.

Automation ought to be provided by the framework. If "driving the
browser" means launch the browser, that requires an application.

>>> * tests run individually, or as a suite
>>> * test code and results are easy to examine
>> I second Garrett's list - here I'd like to be even more specific:
>> * framework should not require adding a lot of code surrounding the one
>> being tested, the same test case that is used for running a test should
>> be usable for debugging, setting breakpoints and stepping through

It should be easy to figure out why a test fails and if YUI Test is
used, it ought to be fixed so that a stack trace is provided, where
possible. That could be toggleable, as in firebug.

FAIL: [+] testSomeFoo

> The tests should also be able to be extracted from the test suite and run in
> implementers' testing frameworks, which calls for a certain degree of
> abstraction.  I've heard many requests for this.
>> This also implies my next point
>> * framework should not require loading several files/frames/scripts for
>> scaffolding - adds complexity to debugging (JSUnit's biggest weakness)
>>> * suite can be paused/resumed at any time
>>> * framework handles asynchronous testing
>>> * events can be created and dispatched easily
>>> * does not use faulty strategies (browser detection, etc).
>> !!!
> Is that a +1?

Probably not. "Faulty strategies" wasn't presented as a pop poll. No
matter how popular they may be, they do not serve a basis for valid
logical assertions.

I'd interpret Hallvord's "!!!" as more like emphasis of that point
than pop-poll +1.

>>> Suite Implementation:
>>> * organized so that it is easy to find tests (searchable, sortable)
>>> * search engine friendly (so that tests can be found)
>>> * links to the relevant specifications, which link back to the tests
> This would be very useful, and I'd love to see this happen.  We might need
> to come up with some convention for marking up the normative requirements in
> the spec itself to facilitate this, which I've started on for DOM3 Events.
>> * It should be simple to exclude specific tests from a test run (i.e.
>> useful for bugs that crash or hang the UA being tested)
>> * It should be trivial to harvest result of the entire suite through a
>> form POST of results in some simple format
>> (I explicitly specify a form POST there because this is the only way to
>> move such a chunk of data across domains. An obvious use case is a
>> browser vendor running the tests located at www.w3.org but storing
>> results from a frontend at browservendor.com/qa/testresults .)
>> * It should give a clear and simple "score" after a test suite run,
>> which will help standards compliance in general because passing the
>> tests and getting a good score will be seen as important. Testing the
>> browser you're using should be as simple as clicking a link, waiting a
>> bit and reading the score.
> This implies automated testing.

Yes, of course.

 I would like to do that as much as
> possible, while keeping in mind that there are some tests that may need to
> be done manually.  The test suite should incorporate both.

I can't think of any tests that need to be done manually and I can't
think of where manual tests preferable.

> Finally, it should be easy for people to create these tests, and to submit
> them for review and incorporation into the official test suite. Creating
> tests is onerous work, and it would go easier with more folks contributing.

For normal projects, the test code never sees production. In this
case, the tests are the production, so it's a bit different.

What if test code could be pulled from a repo and published? Or should
they go into a database.

Selenium Grid launches browsers and uses URLs to send commands.
Results from that could be published to a log, a webapp could grab the
log, and put it in a table that reports results. The results could be
by browsers and versions tested, and/or by feature. A lot of work up

I can take about an hour or so to get a basic example of a DOM 2 test.
I can create and upload a test to a public place for an example.

>>> Results of various browsers and versions could be provided on a
>>> separate page.
>>> YUI Test is open source.
>> ..and a pretty nice framework. I don't remember to what extent it
>> depends on other YUI stuff, it would be my only gripe if it loaded
>> several K of YUI code before getting to the tests. Overall, I'll by far
>> prefer YUI Test to JSUnit. There are alternatives like the MochiKit test
>> framework.
> I'll take a closer look at YUI Test, thanks.
>>> http://www.w3.org/DOM/Test/
>>> I do not personally find that test suite to be useful. Suggestion: Why
>>> don't you post to comp.lang.javascript and ask if anyone finds the
>>> test suite useful?
>> Well, it's mostly useful to UA vendors. I can't imagine any author
>> wanting to use it. However, it does tell us something that none of
>> today's library authors use that framework.

OK: What do the actions of today's library authors tell us?

Received on Monday, 20 July 2009 19:48:12 UTC

This archive was generated by hypermail 2.3.1 : Tuesday, 20 October 2015 10:46:14 UTC