- From: James Graham <jgraham@opera.com>
- Date: Wed, 01 Dec 2010 12:54:34 +0100
- To: Maciej Stachowiak <mjs@apple.com>
- CC: "'public-html-testsuite@w3.org'" <public-html-testsuite@w3.org>
On 11/30/2010 07:57 PM, Maciej Stachowiak wrote: > It seems like the intended way to use your framework would be to wrap > this all in a single test(), which would result in only a single line > output table. It seems to me this would result in a much less useful > test. The current test gives a lot of detail about what it tested in > the output when it passes, and reports exactly what went wrong when > it fails. No, the intended way is pretty much what you did. There is a deliberate (and, I believe useful) distinction between an assertion and a Test. Tests are the atomic unit; in simple cases like your example there is one Test per independent statement in the spec. Asserts are intended to be "subatomic", in the sense that passing a single assert doesn't mean that you have done enough to conform to any part of the spec. A typical example of the use of this distinction is where you have some test that manipulates the dom somehow and verifies the results are what you expect. In this case you might first assert that you ended up with the expected number of nodes, then assert that each node had the right type, tagName, and so on. > I made a monolithic version as an experiment, where everything is > wrapped in one test(), and this has three unfortunate side effects: Yes, that's not designed to work well. > Relatedly, whether or not you use a test() per assert_equals(), > assert_equals() gives a lame error message: > > assert_equals: expected 4 got 1 > > This doesn't include the expression that gave the wrong result. This > is unfortunate when using test-per-assert but damn near useless if > you combine multiple assert_equals in one test. Getting the expression out is non-trivial since there is no standard API to do so. I could add browser-specific code to get at the stack where available. Would that be useful? The alternative — using eval — comes with its own set of problems such as requiring code-as-a-string (breaking syntax highlighting and other editor features) and invoking unusual language edge cases. I don't like that approach in the webkit tests I have seen use it and I would prefer to avoid that road here. > Therefore, I would really appreciate one of the following changes: > > A) Add a test_assert_equals() combo call (perhaps the shorthand > should even take an expression to eval rather than a function, but it > doesn't really matter). That way, you can easily have a list of > assertions each of which is reported independently, with a useful > error message that reports the bad value. So an API improvement to make writing the kind of test in your example easier would be satisfactory? Rather than making specific API for the run-a-test-with-a-single-assert-and-no-other-code case, a possibility would be to have a generate_tests() function. You would pass this a function, and an array containing parameters to pass to the function call e.g. generate_tests(assert_equals, [ [1+1,2], [2+3,5]]) This would generalize better since one could pass in a custom written function that did more than a simple assert_* > B) Report each assertion, not just each test, independently in the > output, and by default do not stop at a failed assertion. (If needed, > we could have variant functions that automatically stop at a bad > result, if it makes no sense to continue; and the test should report > when not all assertions were run). I don't like this idea since it confuses assertions with Tests. I am also not sure in general how one would determine if all assertions were run (excepting hardcoding in the expected number of assertions).
Received on Wednesday, 1 December 2010 11:55:14 UTC