[Bug 3756] Pre-canonicalize the testsuite

http://www.w3.org/Bugs/Public/show_bug.cgi?id=3756





------- Comment #13 from per@bothner.com  2006-09-25 21:47 -------
(In reply to comment #12)
> In part this is because our decision to have implementors
> canonicalize the expected result is one that was made a very long time ago and
> has been found acceptable by many implementors.

The problem is that this can mroe easily hide errors, thus making the testsuite
stricly less powerful.  To verify that matches(ACTUAL,EXPECTED) by doing
match(canonicalize(ACTAL),canonicalize(EXPECTED)) is doing a stricly less
powerful test.  A bug in canonicalize can lead to a false positive. 
Alternatively, if canonicalize "throws away too much information" you can also
get false positives.  E.g. if someone implements canonicalize by taking the
string value of the argument then a test case is more likely to pass even in an
implementation fails to emit correct namespace declarations.

> In part this is also because we
> are about to "declare victory" when we publish XQTS 1.0.1 (containing bug fixes
> for reports that we've received over the last several weeks).

Yes, I understand the timing issue, and I apologize for not brining this issue
up earlier.
> 
> I suggest that if you want to reduce the cost running the test suite
> repeatedly, that you first replace our expected results with their canonic
> representations on your local copy.

I not so concerned about the cost (as Mike Kay said one only needs to do
canonicalization for the relatively few tests that aren't canonicalized), but
it does reduce the "power" of the testsuite, and its usefulness in avoiding
regressions.

Received on Monday, 25 September 2006 21:48:05 UTC