[Bug 13445] New: test-group VersionProlog is a mess

http://www.w3.org/Bugs/Public/show_bug.cgi?id=13445

           Summary: test-group VersionProlog is a mess
           Product: XML Query Test Suite
           Version: 1.0.3
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: XML Query Test Suite
        AssignedTo: benjie.nguyen@gmail.com
        ReportedBy: jmdyck@ibiblio.org
         QAContact: public-qt-comments@w3.org


Checking the CVS history of XQTSCatalog.xml and of the files
in Queries/XQuery/Expressions/PrologExpr/VersionProlog/,
I find an alarming amount of churn, all revolving around
the following paragraph in 4.1 [Version Declaration]:

    If a version declaration is present, no Comment may occur
    before the end of the version declaration. If such a Comment
    is present, the result is implementation-dependent.

(I refer to this as "the comment-before-versiondecl clause".)

I've appended the whole history below. The upshot is:

(1) The test cases that originally (or nearly originally) were intended to
    address this clause no longer do so.

(2) For test cases
        version_declaration-001
        version_declaration-002
        version_declaration-003
        version_declaration-004
        K-VersionProlog-1
    the catalog says
        <expected-error>*</expected-error>
    which is inconsistent with the query.

So:

Is there any point having test cases with comment-before-versiondecl,
when the resulting behaviour is implementation-dependent? (Do the
results tell us anything?)

If yes, then we should undo most of the changes to all the
version_declaration-nnn test cases.  If no, we should probably
just remove them.


-----------------------------------------------------------------

Abbreviations:
    vd      = version_declaration
    pv      = prolog-version
    KVP     = K-VersionProlog
    error=* = <expected-error>*</expected-error>
    TTF     = Testing Task Force


2005-10-05:
Ravi Chennoju (rchennoj) adds test cases vd-001 through vd-009
(XQTSCatalog.xml rev 1.349):
    001-005 expect no error
    006-008 expect XPST0031
    009    expects XPST0003
All have a comment preceding the VersionDecl, and so the behaviour
is implementation-defined according to the XQuery spec, but Ravi is
presumably not aware of this.

2005-11-01:
Ravi adds XQST0038 to vd-005's expectations.
(XQTSCatalog.xml rev 1.389)

2006-02-06:
Carmelo Montanez (cmontane) adds vd-010, expects XQST0087.

2006-02-15/17:
Apparently, Carmelo becomes aware of the comment-before-versiondecl
clause, because he adds pv-1 thru pv-12, all of which have
comment-AFTER-versiondecl. In fact, pv-4 through pv-12 are simply
copies of vd-001 through vd-009 with the versiondecl moved up.
At about the same time, he adds 
    <expected-error>*</expected-error>
to vd-001 through vd-010 (XQTSCatalog.xml rev 1.522), presumably to
reflect their implementation-dependent behaviour.

2006-05-11:
In Bug 3275, Jinghao Liu thinks vd-1 shouldn't expect error=*.  Andrew
and Ravi agree that it's there because of the comment-before-versiondecl
clause.

[So, the intent of the tests seems fairly clear: test cases vd-*
exercise the imp-dep behaviour of comment-before-versiondecl, and
pv-* exercise the non-imp-dep behaviour that vd-* were originally
intended to cover. Call this the Revised Intent. ]

---

2006-05-22:
On the TTF mailing list, Franz Englich (fenglich) rightly points out
that vd-005 thru vd-010 expect both a specific error code and error=*,
which is redundant. In each case, one or the other should be removed.
http://lists.w3.org/Archives/Member/member-query-test/2006May/0045.html

2006-05-22:
cmontane removes error=* from vd-009 (XQTSCatalog.xml rev 1.603)

2006-06-08:
TTF meeting 64 decides that in each case, error=* should be removed:
> ACTION TTF-064-03: Carmello to fix version_declaration-010 test and
> remove '*' for expected error and use specific error
> ACTION TTF-064-04: Ravi to fix version_declaration-00[5,6,7,8] test
> and remove '*' for expected error and use specific error
http://lists.w3.org/Archives/Member/member-query-test/2006Jun/0054.html

(That is, it seems the TTF forgot that error=* was there to cover the
imp-dep behaviour of comment-before-versiondecl. As far as I can see,
they should have decided to remove the specific error.)

2006-06-08: cmontane removes error=* from vd-010 (rev 1.618)
2006-06-09: rchennoj removes error=* from vd-005 thru 008 (rev 1.619)

(This leaves vd-001 thru 004 expecting a value or error=*, still in
line with the Revised Intent, but vd-005 thru 010 expecting specific
errors, which is incorrect, and will bite us later.)

---

(2006-07-19: cmontane moves vd-005)

(2006-09-15:
In Bug 3731, Per Bothner questions error=* for vd-001 thru 005, but he's
just misreading the metadata.)

2007-01-26
In Bug 4281, Michael Kay points out the comment-before-versiondecl in
vd-010, which is inconsistent with its specific error message.

2007-07-23
In vd-010, Andrew moves up the versiondecl, making it consistent with
the error message, though inconsistent with the Revised Intent.
(2007-08-09: In the Bug, Andrew reports making this change.)

2007-11-22
fenglich adds KVP-1 through KVP-5, all with comment-before-versiondecl.
The comment for KVP-1 clearly indicates that this is intentional, but oddly,
only KVP-1 has error=*; the others expect a specific error or a value.

2009-05-20
In Bug 6934, Josh Spiegel lists many cases of comment-before-versiondecl,
including vd-* and KVP-*.  His implementation, which (conformantly) raises
an error in such cases, only passes vd-001 thru 0004.

2009-10-14
To "fix" bug 6934, fenglich moves up the versiondecl in vd-001 thru 008, and
KVP-1 through 5. However:

    vd-001 thru 004 and KVP-1 still expect error=*,
    which is inconsistent with their (changed) queries.

    None of vd-* adhere to the Revised Intent any more.
    That is, there are no longer any tests that exercise the
    comment-before-versiondecl clause. The vd-* queries
    are basically duplicates of pv-* queries. 
    (But missing a couple changes that some pv-queries received.)

xxx

-- 
Configure bugmail: http://www.w3.org/Bugs/Public/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

Received on Friday, 29 July 2011 06:46:48 UTC