[Bug 19782] test cases using 'assert-eq' for numeric values instead of 'assert'

https://www.w3.org/Bugs/Public/show_bug.cgi?id=19782

--- Comment #6 from Sorin Nasoi <spungi@gmail.com> ---
(In reply to comment #5)
> 1. If you have enumerated all the affected tests then it would be helpful to
> supply the list. It would be even better to supply a shorter list where the
> changes are essential, since resources are limited.
> 
> 2. I don't understand comment #4. What's wrong with the test as written, and
> how does it relate to this bug report?
Let me try to answer both issues in the same time because they are connected:
I implemented the function for checking the "assert-eq" into a FOTS driver.
I run all the tests in the FOTS that have this assertion defined and I analysed
the failures.

I did not find any way of implementing an "assert-eq" check that is able to
account for all the misuses of this assertion into the FOTS.

1) if I make the assumption that the "assert-eq" was created to compare the
string values of the actual and expected results, than a bunch of tests fail.

Here are some examples:
<result><assert-eq>1.5707963267948966e0</assert-eq></result> compared to an
actual result of 1.5707963267948966
<result><assert-eq>0e0</assert-eq></result> compared to an actual result of 0
<result>"X"</assert-eq></result> compared to an actual result X (because of the
additional commas in the expected result)

2) if I make the assumption that the "assert-eq" was created to compare the
values of the actual and expected results but only after casting them both to
the type of the actual result another set of tests fail.
Some of the above mentioned tests pass, because 
xs:double(1.5707963267948966e0) eq xs:double(1.5707963267948966) and also
xs:double (0e0) eq xs:double(0)

IMHO one of the main causes for these confusions (both for those writing tests
and for those trying to write drivers that are able to run the tests correctly)
is the lack of information in the FOTS catalog for the "assert-eq":

"assert-eq  denotes an element whose assumption when run must equal the
expected test result."

IMHO we should give more specific details about how the result and actual
results are compared:
- is there any serialization done to the actual result?
- the things that are compared are the string values or should there be some
casting done first?

Also some common sense advices would also be very helpful, something like:
- if one is interested in checking the type of the results please consider
using "assert-type" instead of "assert-eq"
- if one needs to compare numeric values that have decimals please consider
using "assert" instead of "assert-eq".

So here is what I would propose:
- please add the necessary details into he FOTS schema for "assert-eq"
- after I implement the "assert-eq" in the driver it would be very simple for
me to provide a verbose list of all tests in the FOTS that misuse "assert-eq"
(according to the complete description from the FOTS schema).

Would this work?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

Received on Wednesday, 31 October 2012 10:16:15 UTC