[Bug 11446] Provide alternate result for test K2-Literals-1

http://www.w3.org/Bugs/Public/show_bug.cgi?id=11446

--- Comment #2 from Sorin Nasoi <spungi@gmail.com> 2010-12-01 10:19:25 UTC ---
Thanks for the quick reply, here are my thoughts:

I agree with the point you are making, and I am not trying to "beat our heads
against a brick wall", I am trying to make the XQTS better by reporting some
discrepancies that I found.
I think you agree that feedback is a good thing, right?

So,the 
http://dev.w3.org/2006/xquery-test-suite/PublicPagesStagingArea/Guidelines%20for%20Running%20the%20XML%20Query%20Test%20Suite.html
section "Comparing Results" state that:
"Error: The expected result of the test case is an error, identified as an
eight-character error code (e.g., XPST0003). The result of a test is true if
the implementation raises an error. However, raising an error because an
implementation does not support the feature is not considered a correct
result."

This can be improved because:
1- "The result of a test is true if the implementation raises an error."
This is vague and because of this too permissive IMHO.

2- the part "However, raising an error because an implementation does not
support the feature is not considered a correct result."
is wrong:
XQTS also has tests for XQuery 1.1 features. These tests accept raising an
error if the implementation does not support the feature when the
implementation state that it supports XQuery 1.0, correct?

Instead, let's add the suggestion you made:
"If the expected result of a test is a static error, a test harness may report
the test as passing no matter the static error code it returns. 
In the case of dynamic errors, the return error code must match one of the
expected error codes of the test."

Does this makes sense?

-- 
Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

Received on Wednesday, 1 December 2010 10:19:27 UTC