W3C home > Mailing lists > Public > public-html-testsuite@w3.org > December 2010

Re: Request for Feedback On Test Harness

From: James Graham <jgraham@opera.com>
Date: Wed, 01 Dec 2010 12:54:34 +0100
Message-ID: <4CF6377A.1000506@opera.com>
To: Maciej Stachowiak <mjs@apple.com>
CC: "'public-html-testsuite@w3.org'" <public-html-testsuite@w3.org>
On 11/30/2010 07:57 PM, Maciej Stachowiak wrote:


> It seems like the intended way to use your framework would be to wrap
> this all in a single test(), which would result in only a single line
> output table. It seems to me this would result in a much less useful
> test. The current test gives a lot of detail about what it tested in
> the output when it passes, and reports exactly what went wrong when
> it fails.

No, the intended way is pretty much what you did. There is a deliberate 
(and, I believe useful) distinction between an assertion and a Test. 
Tests are the atomic unit; in simple cases like your example there is 
one Test per independent statement in the spec. Asserts are intended to 
be "subatomic", in the sense that passing a single assert doesn't mean 
that you have done enough to conform to any part of the spec. A typical 
example of the use of this distinction is where you have some test that 
manipulates the dom somehow and verifies the results are what you 
expect. In this case you might first assert that you ended up with the 
expected number of nodes, then assert that each node had the right type, 
tagName, and so on.

> I made a monolithic version as an experiment, where everything is
> wrapped in one test(), and this has three unfortunate side effects:

Yes, that's not designed to work well.

> Relatedly, whether or not you use a test() per assert_equals(),
> assert_equals() gives a lame error message:
>
> assert_equals: expected 4 got 1
>
> This doesn't include the expression that gave the wrong result. This
> is unfortunate when using test-per-assert but damn near useless if
> you combine multiple assert_equals in one test.

Getting the expression out is non-trivial since there is no standard API 
to do so. I could add browser-specific code to get at the stack where 
available. Would that be useful? The alternative  using eval  comes 
with its own set of problems such as requiring code-as-a-string 
(breaking syntax highlighting and other editor features) and invoking 
unusual language edge cases. I don't like that approach in the webkit 
tests I have seen use it and I would prefer to avoid that road here.

> Therefore, I would really appreciate one of the following changes:
>
> A) Add a test_assert_equals() combo call (perhaps the shorthand
> should even take an expression to eval rather than a function, but it
> doesn't really matter). That way, you can easily have a list of
> assertions each of which is reported independently, with a useful
> error message that reports the bad value.

So an API improvement to make writing the kind of test in your example 
easier would be satisfactory? Rather than making specific API for the 
run-a-test-with-a-single-assert-and-no-other-code case, a possibility 
would be to have a generate_tests() function. You would pass this a 
function, and an array containing parameters to pass to the function 
call e.g.

generate_tests(assert_equals, [
[1+1,2],
[2+3,5]])

This would generalize better since one could pass in a custom written 
function that did more than a simple assert_*

> B) Report each assertion, not just each test, independently in the
> output, and by default do not stop at a failed assertion. (If needed,
> we could have variant functions that automatically stop at a bad
> result, if it makes no sense to continue; and the test should report
> when not all assertions were run).

I don't like this idea since it confuses assertions with Tests. I am 
also not sure in general how one would determine if all assertions were 
run (excepting hardcoding in the expected number of assertions).
Received on Wednesday, 1 December 2010 11:55:14 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Wednesday, 1 December 2010 11:55:14 GMT