[Bug 22423] New: total count of 'fail' test-cases incorrect in the XSL stylesheet

https://www.w3.org/Bugs/Public/show_bug.cgi?id=22423

            Bug ID: 22423
           Summary: total count of 'fail' test-cases incorrect in the XSL
                    stylesheet
    Classification: Unclassified
           Product: XPath / XQuery / XSLT
           Version: Working drafts
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: XQuery 3 & XPath 3 Test Suite
          Assignee: oneil@saxonica.com
          Reporter: spungi@gmail.com
        QA Contact: public-qt-comments@w3.org

As far as I understand, a test-case should be considered 'fail' if:
- it is reported as a FAIL
- a test-case that is present in the test suite for which no result has been
reported

Yet a test-case reported as 'notRun' with an additional comment attribute
specifying the reason for which the test-case was not run counts as 'fail' in
the reports generated by the XSL transformation.

Here is an example:
If one reports 
<fots:test-case name="fn-collection-4" result="notRun" comment="Default
collection is always an empty sequence."/>
the test-case counts as 'fail' in the reports although having a way of setting
default collection to something different than () is *not* a required
functionality.

I see 2 approaches to fix this:
1) remove the 'notRun' status altogether and add new dependencies for the
test-cases that require functionality outside let's say "minimal conformance".
This way the test-cases would be reported as pass/fail or n/a (meaning a
dependency was not met).
Because, IMHO, having a way of setting additional collations or setting the
default collection to something different than () should be treated as a
dependency.

As a result, the following paragraph would be remove from "Running QT Test
Suite":
"Also: XQuery and XPath do not explicitly require implementations to provide a
way for all values in the static and dynamic context to be set. For example, an
implementation that does not allow any collations to be defined (other than the
Unicode Codepoint Collation) would be technically conformant; so also would be
an implementation in which the default collection is always an empty sequence.
There is therefore an implicit dependency, when values are set in the
environment, that the implementation allows such values to be set. An
implementation that cannot set the context as required should behave in the
same way as if a dependency has not been satisfied, ideally reporting the test
as "notRun" with a reason."

2) update XSL stylesheet to count as 'fail' only test-cases reported as
'notRun' and do not have a comment attribute stating the reason why they were
not run.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

Received on Friday, 21 June 2013 10:28:53 UTC