Review of F&O LCWD against QAWG Spec Guidelines

On behalf of the Quality Assurance Working Group and Interest Group, I
am sending a review of the F&O Last Call Working Draft against the LCWD
of W3C Specification Guidelines found at 
http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/ 
for which processing of Last Call comments is nearly complete. (The F&O
editors were given this review earlier. This mailing to ensure that it's
visible on the public list.)

The essence of the Spec Guidelines is the same now as it was in the
above-cited draft, while some verbiage has been adjusted. The QAWG
considers this review a good approximation of the spec reviews that
should be standard procedure when the Spec Guidelines go to Rec status.
I hope that this review will inspire the editors of other documents
being issued by the XSL and XQuery Working Groups.
.................David Marston

=============== REVIEW ==================== 
Review of XQuery 1.0 and XPath 2.0 Functions and Operators
Last Call Working Draft 02 May 2003
http://www.w3.org/TR/2003/WD-xpath-functions-20030502/
Criteria: SpecGL of 10 February 2003
Reviewed by David Marston; completed 30 June 2003

Overview
The document describes measurable behavior required from a set of
functions and operators (F&O). The F&O are intended to be hosted by
XQuery or XPath, and XPath itself is hosted by other facilities.
Thus, the conformance tests would be applied against a processor for 
XQuery, XSLT, XPointer, or XForms (and other classes of product to 
come). There is no statement that the F&O would be a W3C-wide standard, 
but that could happen, just as XML Schema Part 2 is a W3C-wide standard
for datatypes.

Several Dimensions of Variability apply to F&O. The "classes of product"
are XQuery and the applications that host XPath. It's unclear whether
there is a policy about implementing a subset of the functions, though
there is no mention of profiles or modules that would suggest subsetting
is allowed. The policy on extensions is not explicit. Additionally,
deprecation and discretionary items are used. The document needs a
conformance section to gather all the conformance information. Some of
the suggested content of this section will be found in the analysis by
checkpoint below.

Guidelines and Checkpoints
+++Guideline 1. Define Scope.
Overview: the double indirection from F&O to XPath to applications that
host XPath presents some interesting complications. The draft makes a
good approximation of defining the scope; minor reworking will satisfy
this guideline.

1.1 Include the scope of the specification. [Priority 1]
Satisfied by verbiage in 1.5 and 1.6.

1.2 Illustrate what is in scope [Priority 2]
Not satisfied. In particular, there should probably be a statement that
every host application may accept additional functions (and operators?)
that may be defined by the other W3C specs, industry consortia, vendors'
implementations or the user. The scope of F&O appears to be a core set
of functions, but not the entire set for any application.

1.3 Provide Usage Scenarios. [Priority 3]
Not satisfied, but could be (IMHO) by a simple statement that the
hosting application (XQuery, XSLT, etc.) must provide the usage
scenarios for expressions in general.

1.4 Provide examples. [Priority 2]
Satisfied.

+++Guideline 2. Identify what needs to conform and how.
Overview: as with Guideline 1, the draft is close to satisfying this.

2.1 Identify all classes of product. [Priority 1]
Part 1.5 should be strengthened to address whether any W3C spec that
currently requires XPath 1.0 will, upon upgrading to require XPath
2.0 in the future, automatically require all of F&O. The WG should
consider a statement about whether other specs with a non-XPath
expression language can use F&O. The WG should address whether XPath
1.0 can be mixed with F&O or must be limited to the XPath 1.0 set of
functions.

2.2 For each class of product, define the conformance requirements.
   [Priority 1]
This should be stated more clearly, either in 1.5 or (better) in a
new conformance section. Do XQuery and XPath differ in any way with
respect to F&O conformance, like functions behaving differently?

2.3 Identify which of the categories of object are specified in the
   document as a whole. [Priority 3]
Satisfied by material in chapter 1.

2.4 If there are several classes of products, define their relationships
   and interaction with other dimensions of variability. [Priority 2]
Not satisfied, but could easily do so by stating that XPath has
deprecation requirements and XQuery doesn't.

+++Guideline 3. Specify conformance policy.
Overview: the draft needs to address the subsetting question.

3.1 Specify any universal requirements for minimum functionality.
   [Priority 2]
Not satisfied. There should be a clear statement about whether the
entire set of functions must be implemented. Does the rule vary by
class of product? Under what circumstances, if any, can implementers
choose to not implement certain functions? Can they ever choose to
not implement certain operators?

3.2 If special conformance terms are used, include a definition in the
   specification. [Priority 1]
Satisfied by 1.1.

3.3 Justify any usage of a dimension of variability [Priority 1]
Not satisfied. This could appear in the conformance section.

+++Guideline 4. Address the use of profiles to divide the technology.
Overview: apparently, there are no profiles.

4.1 Indicate whether or not the use of profiles is mandatory for
   conformance. [Priority 1]
Not satisfied, but could easily do so by stating that the host
application may have profiles, but F&O doesn't spawn any new
profiles.

4.2 If profiles are chosen, define any minimal requirements. [Priority 2]
Not applicable.

4.3 If profiles are chosen, define their relationships and interaction
   with other dimensions of variability. [Priority 2]
Not applicable.

4.4 If profiles are chosen, address rules for profiles. [Priority 2]
Not applicable, unless (A) subsetting of the functions is allowed,
and (B) the WG intends to control the subsetting. For example,
the WG could specify a profile for date/time awareness and say that
a certain handful of functions must either all be implemented to
meet the profile or all be unimplemented (raise errors) if not.

+++Guideline 5. Address the use of modules to divide the technology.
Overview: apparently, there are no modules.

5.1 If modules are chosen, indicate any mandatory conditions or
   constraints on their usage. [Priority 1]
Not applicable.

5.2 If modules are chosen, define their relationships and interaction
   with other dimensions of variability. [Priority 2]
Not applicable.

+++Guideline 6. Address the use of functional levels to divide the
  technology.
Overview: apparently, there are no levels.

6.1 If levels are used, define their relationships and interaction with
   other dimensions of variability. [Priority 2]
Not applicable.

+++Guideline 7. Identify the relation between deprecated features and
  conformance.
Overview: the topic of backward compatibility arises in a few places. In
each case, the old feature should get an indication of its desirability
for future use. Of special interest are the extra datatypes that may be
deprecated in the "xdt" namespace if they are added to Schema Part 2.
The appendix on compatibility could be expanded to include any and all
deprecation concerns.

7.1 Identify each deprecated feature. [Priority 1]
Not satisfied. Need to discuss former F&O namespaces (presumably not
recognized), and acceptance of "Infinity" and "-Infinity" (vs. "INF"
and "-INF") in expressions. Other deprecated features can be found by
searching for "backward compatibility", but it would be helpful to have
a consolidated list. Expect that test case creators will want to refer
to these features by name.

7.2 For each class of product, specify the degree of support required for
   each deprecated feature and the conformance consequences of the
   deprecation. [Priority 1]
Not satisfied. May need to let XPath actually contain the policy,
but F&O should state how to ascertain the policy.

7.3 If deprecation is used, define its relationships and interaction with
   other dimensions of variability. [Priority 2]
Not satisfied. Is backward compatibility always required for XPath?
Can XQuery have XPath-1.0-compatibility mode? Is there any policy
regarding user-defined types that may vary due to deprecation?

7.4 Include an explanation for the deprecation. [Priority 3]
Satisfied by verbiage in parts 2.3, 9.2 (note about possible
change of namespace), and 15.1.4. This fulfills the checkpoint 3.3
requirements above with respect to deprecation.

7.5 Include examples to illustrate how to avoid using deprecated features.
   [Priority 3]
Satisfied for boolean() function. Not satisfied where "Infinity"
vs. "INF" is discussed. (At a minimum, give an example of an IfEXpr
used to change the strings.)

+++Guideline 8. Define discretionary items.
Overview: this is a dimension that cries out for better-organized
information. There are items marked "implementation defined" or
"implementation dependent" or "may" (or otherwise open to variance) in
parts 2, 4, 5, 6, 7, 9, 15, 16, and 17. The editors should consider
the point-of-view of test case creators as they improve coverage of
this topic.

8.1 State the circumstances for when discretionary items are allowed
   [Priority 1]
Satisfied in some places, but not in 15.1.15, 15.4.5 (schemes), and
17.12 (is this discretionary?).

8.2 For implementation dependencies, address the allowable differences
   between implementations [Priority 1]
Satisfied, but the verbiage is spread around. A consolidated list with
item names will help test case creators.

8.3 Indicate any constraints on the number of choices/options that can
   be implemented for discretionary choices [Priority 2]
If the WG intends to allow implementations to support the codepoint
collation but not the provided URI to name it, then support of that
URI should be presented as a yes/no discretionary choice. (On the
pro forma, the question would be: "Do you recognize
http://www.w3.org/2003/05/xpath-functions/collation/codepoint
as a name for collation based on Unicode codepoint values?") If the 
WG intended to require that this URI be recognized by all F&O
implementations, section 7.3 should say so. Is there any scheme that
fn:doc() MUST or SHOULD support (e.g., file)? Most other items are
satisfactory.

8.4 Promote consistent handling of discretionary choices. [Priority 2]
Not satisfied. Among the areas where consistency could be enforced
are: collation names (consistency across all functions), URI schemes
(consistency between fn:doc and fn:collection), and min/max.

8.5 If discretionary items are used, define their relationships and
   interaction with other dimensions of variability. [Priority 2]
Not satisfied. Can the range of choices on any item vary according
to class of product (host application), support of a partial set of
the functions (if allowed), support of deprecated features, or
extensions (if allowed)?

+++Guideline 9. Allow extensions or NOT!
Overview: this area should get cleared up quickly if the WG agrees that
all other functions must be in different namespaces.

9.1 Indicate if the specification is extensible. [Priority 1]
Not satisfied, but could easily do so by adding statements to 1.5
that no other functions can be placed in the F&O namespace, but that
host application may add more built-in functions in other namespaces.
The verbiage for this checkpoint could instead be in a conformance
section, if one is added.

9.2 If extensions are allowed, define their scope and constraints.
   [Priority 1]
Not applicable, presuming no functions can be added to the F&O
namespace.

9.3 Prevent extensions from contradicting the specification. [Priority 1]
Should be satisfied by the repairs to part 1.5 that satisfy checkpoint
9.1 above.

9.4 Define a uniform mechanism to create an extension [Priority 3]
Should be satisfied by suggested verbiage regarding additional
functions supplied as built-ins by host, plus acknowledging that
host may support user-defined functions in a non-W3C namespace.
(Also see 9.5 below.)

9.5 Require that extensions be published. [Priority 3]
Probably not applicable, but it would be good to have statements
that a vendor can have additional built-in functions in a non-W3C
namespace and can implement a function package from another
consortium (e.g., EXSLT) in a non-W3C namespace. The latter is an
example of a "published extension" as contemplated in this
checkpoint.

9.6 Require that implementations provide interoperable alternatives to
   extensions [Priority 3]
Not satisfied.

9.7 If extensions are allowed, define their relationships and
   interaction with other dimensions of variability [Priority 2]
Not applicable, presuming no functions can be added to the F&O
namespace.

+++Guideline 10. Provide a conformance clause.
Overview: as mentioned at the top of this review, addition of a
conformance section is the central activity that addresses most of
the identified shortcomings for both A-level and AA-level
conformance to these guidelines.

10.1 Include a conformance section. [Priority 1]
Not satisfied. There is a specialized conformance section at 9.1.1,
but no general gathering point for conformance information.

10.2 Make normative reference to specifications on which the current
    specification depends [Priority 2]
Satisfied, more or less, by Appendix A.1 and certain verbiage in the
beginning of chapter 1.

+++Guideline 11. Specify how to make conformance claims.
Overview: this material is not in F&O now, but the requirements are
only indirect as long as F&O is not implementable in standalone
fashion.

11.1 Identify and define all conformance designations. [Priority 1]
Not satisfied. The conformance section should allude to the
conformance designations of the host application. In particular, if an
XPath host application is allowed to treat backward compatibility as an
optional module, then there would be conformance implications for F&O.

11.2 Provide specific wording of the claim. [Priority 3]
Not applicable (directly). The host application needs this verbiage.

11.3 Provide a conformance disclaimer. [Priority 3]
Not satisfied, though possibly not applicable if the WG thinks that
provided disclaimers for the host apps would suffice to cover F&O.

11.4 Impose no restrictions about who can make a claim or where claims
    can be published. [Priority 1]
Not applicable (directly).

+++Guideline 12. Publish an Implementation Conformance Statement
  proforma.
Overview: nothing has been done here. Constructing a master list of
permitted variability (along 3-4 dimensions) may set the stage for a
proforma questionnaire.

12.1 Provide an Implementation Conformance Statement proforma.
    [Priority 3]
Not satisfied. See discussion at 8.3 above for an example of an 
item that is specific to F&O conformance. Buried in the "conformance
note" in 9.1.1 is a requirement that the implementation document one
particular piece of information that belongs in the proforma. Most of
the discretionary items could use one or more items in the proforma
to stimulate vendors to document details that may vary among
implementations.

12.2 Require the ICS be completed as part of the conformance claim.
    [Priority 3]
Not satisfied.

+++Guideline 13. Clearly identify conformance requirements.
Overview: the major item here is the conformance section or navigation
mechanism that draws all conformance policies and variability together.

13.1 Use conformance key words. [Priority 1]
Not satisfied. Part 1.1 has its own definitions of "may" and "must"
and no explanation of why RFC 2119 is not referenced.

13.2 Distinguish normative and informative text. [Priority 1]
Partly satisfied. All example parts (1.4.1, 3.1, etc.) should say
"(non-normative)" in their title.

13.3 Use consistent terminology. [Priority 1]
Appears to be satisfied.

13.4 Provide a fast way to find conformance information [Priority 2]
Not satisfied. In particular, discretionary items appear in about
20 separate places, and deprecation is discussed in three.

+++Guideline 14. Provide test assertions.
Overview: this area is not addressed. To the extent that the function
signatures, as presented in their special style, can define some
constraints, there is a start. Also, testable sentences abound. (For
example, "If the second argument is an empty sequence, an empty sequence
is returned.") If the WG is interested in pursuing development of test
assertions, they should designate a member to interact with QAWG and
possibly negotiate about an experiment.

14.1 Provide test assertions [Priority 2]
Not satisfied.

14.2 Provide a mapping between the specification and the test
    assertions list [Priority 2]
Not satisfied.

Scorecard
Of the Priority 1 criteria, 3 are satisfied (1.1, 3.2, 13.3), 7 almost
satisfied (2.2, 4.1, 7.2, 8.1, 9.1, 9.3, 13.2), 6 not satisfied (2.1,
3.3, 7.1, 10.1, 11.1, 13.1), 1 barely satisfied (8.2), 2 are not
applicable (5.1, 11.4), and 1 is contingently not applicable (9.2).

Of the Priority 2 criteria, 2 are satisfied (1.4, 10.2), 2 almost
satisfied (2.4, 8.3), 8 not satisfied (1.2, 3.1, 7.3, 8.4, 8.5, 13.4,
14.1, 14.2), 4 are not applicable (4.2, 4.3, 5.2, 6.1), and 2 are
contingently not applicable (4.4, 9.7).

Of the Priority 3 criteria, 2 are satisfied, 2 almost satisfied, 5 not
satisfied, 1 is not applicable, and 1 is contingently not applicable.

Wrap-up
If a new conformance section is added to chapter 1, with content as
suggested above, then F&O will be very close to A-level conforming to
the Spec Guidelines. Most other A-level issues are small. Cleanup of
the small issues should clarify the "contingently not applicable" items.

Even if the WG is only aiming for A-level, they should consider the easy
cleanup of the "almost satisfied" (2.4, 8.3, 1.3, 9.4) and "contingently
not applicable" (4.4, 9.7, 9.5) items, to provide additional benefit.

Achievement of AA-level conformance may seem daunting, but that is
mainly due to the requirements for test assertions. Several items that
are not satisfied could be addressed as a by-product of creating the
conformance section, documenting general conformance policies, and
assembling the list of all discretionary items.

Received on Thursday, 3 July 2003 10:58:51 UTC