Re: Testing framework requirements

On Sun, Jul 19, 2009 at 11:54 PM, Doug Schepers<schepers@w3.org> wrote:
> Hi, Garrett, Hallvord-
>
> Thanks for the helpful comments!
>
> Comments inline...
>
> Hallvord R. M. Steen wrote (on 7/17/09 10:07 AM):
>>
>> On Thu, 16 Jul 2009 09:28:16 +0200, Garrett Smith
>> <dhtmlkitchen@gmail.com> wrote:
>>
>>>> What are some of the desired requirements for an automated testing
>>>> framework
>>>> for testing HTML5/DOM specs from the JavaScript programmatic side?
>>>
>>> Script Testing Framework Features:
>>> * runs in a browser
>
> Runs in any browser, obviously.
>

How would you run a javascript-driven test runner in a browser that
does not have an implementation of ECMAScript?

I would guess some browsers, such as Blackberry or Opera Mobile, may
present some challenges for an automation test.

> I wouldn't preclude the idea that there could be an additional testing
> application that drives the browser and automates the testing.
>

Automation ought to be provided by the framework. If "driving the
browser" means launch the browser, that requires an application.

>
>>> * tests run individually, or as a suite
>>> * test code and results are easy to examine
>>
>> I second Garrett's list - here I'd like to be even more specific:
>>
>> * framework should not require adding a lot of code surrounding the one
>> being tested, the same test case that is used for running a test should
>> be usable for debugging, setting breakpoints and stepping through
>

It should be easy to figure out why a test fails and if YUI Test is
used, it ought to be fixed so that a stack trace is provided, where
possible. That could be toggleable, as in firebug.

FAIL: [+] testSomeFoo

> The tests should also be able to be extracted from the test suite and run in
> implementers' testing frameworks, which calls for a certain degree of
> abstraction.  I've heard many requests for this.
>
>
>> This also implies my next point
>> * framework should not require loading several files/frames/scripts for
>> scaffolding - adds complexity to debugging (JSUnit's biggest weakness)
>>
>>> * suite can be paused/resumed at any time
>>> * framework handles asynchronous testing
>>> * events can be created and dispatched easily
>>> * does not use faulty strategies (browser detection, etc).
>>
>> !!!
>
> Is that a +1?
>

Probably not. "Faulty strategies" wasn't presented as a pop poll. No
matter how popular they may be, they do not serve a basis for valid
logical assertions.

I'd interpret Hallvord's "!!!" as more like emphasis of that point
than pop-poll +1.

>
>>> Suite Implementation:
>>> * organized so that it is easy to find tests (searchable, sortable)
>>> * search engine friendly (so that tests can be found)
>>> * links to the relevant specifications, which link back to the tests
>
> This would be very useful, and I'd love to see this happen.  We might need
> to come up with some convention for marking up the normative requirements in
> the spec itself to facilitate this, which I've started on for DOM3 Events.
>
>
>> * It should be simple to exclude specific tests from a test run (i.e.
>> useful for bugs that crash or hang the UA being tested)
>> * It should be trivial to harvest result of the entire suite through a
>> form POST of results in some simple format
>>
>> (I explicitly specify a form POST there because this is the only way to
>> move such a chunk of data across domains. An obvious use case is a
>> browser vendor running the tests located at www.w3.org but storing
>> results from a frontend at browservendor.com/qa/testresults .)
>>
>> * It should give a clear and simple "score" after a test suite run,
>> which will help standards compliance in general because passing the
>> tests and getting a good score will be seen as important. Testing the
>> browser you're using should be as simple as clicking a link, waiting a
>> bit and reading the score.
>
> This implies automated testing.

Yes, of course.

 I would like to do that as much as
> possible, while keeping in mind that there are some tests that may need to
> be done manually.  The test suite should incorporate both.

I can't think of any tests that need to be done manually and I can't
think of where manual tests preferable.

> Finally, it should be easy for people to create these tests, and to submit
> them for review and incorporation into the official test suite. Creating
> tests is onerous work, and it would go easier with more folks contributing.
>

For normal projects, the test code never sees production. In this
case, the tests are the production, so it's a bit different.

What if test code could be pulled from a repo and published? Or should
they go into a database.

Selenium Grid launches browsers and uses URLs to send commands.
Results from that could be published to a log, a webapp could grab the
log, and put it in a table that reports results. The results could be
by browsers and versions tested, and/or by feature. A lot of work up
front.

I can take about an hour or so to get a basic example of a DOM 2 test.
I can create and upload a test to a public place for an example.

>
>>> Results of various browsers and versions could be provided on a
>>> separate page.
>>>
>>> YUI Test is open source.
>>
>> ..and a pretty nice framework. I don't remember to what extent it
>> depends on other YUI stuff, it would be my only gripe if it loaded
>> several K of YUI code before getting to the tests. Overall, I'll by far
>> prefer YUI Test to JSUnit. There are alternatives like the MochiKit test
>> framework.
>
> I'll take a closer look at YUI Test, thanks.
>
>
>>> http://www.w3.org/DOM/Test/
>>>
>>> I do not personally find that test suite to be useful. Suggestion: Why
>>> don't you post to comp.lang.javascript and ask if anyone finds the
>>> test suite useful?
>>
>> Well, it's mostly useful to UA vendors. I can't imagine any author
>> wanting to use it. However, it does tell us something that none of
>> today's library authors use that framework.
>

OK: What do the actions of today's library authors tell us?

Garrett

Received on Monday, 20 July 2009 19:48:12 UTC