Re: new draft "Framework: Process & Operational Guidelines"

>    This document is a Working Draft, made available by the [18]W3C
>    Quality Assurance (QA) Activity for discussion on the QA email list.

a bit picky but it's "for discussion on the QAWG mailing list."

>    All review comments against the previous version
>    have been accepted and implemented, except as indicated in an
>    associated [19]disposition of comments document. This version still

The disposition of comments actually lists a bunch of "taken"
comments, so it's unclear if it is not up-to-date or if the
description here is wrong.


>    This document is part of a family of QA Framework documents designed
>    to improve the quality of W3C specifications as well as their
>    implementation by solidifying and extending current quality practices
>    within the W3C. The QA Framework documents are:
>      * Introduction
>      * Specification Guidelines
>      * Process and Operational Guidelines (this document)
>      * Technical Guidelines

I suggest we use the same order as usual
 
>    Not only
>    are the WGs the consumer of these guidelines, they are also key

consumers

>    Although not explicitly stated, the W3C Process Document supports the
>    development of conformance test materials.
>     1. "[.] groups may produce technical reports, review the work of
>        other groups, develop sample code or test suites, etc." (sec. 3.)
>     2. "W3C should make every effort to maintain its Recommendations
>        (e.g., by tracking errata, providing test bed applications,
>        helping to create test suites, etc.)" (sec4.)
>     3. In an effort to meet these suggestions and address the
>        implementation requirements of the Process Document, some Working
>        Groups have included the development of conformance materials as
>        part of their CR-exit and PR-entrance criteria (e.g., SVG, UAAG,
>        XSL-FO). This makes sense, since it is natural for test suites and
>        implementations to develop in parallel - each is a help to the
>        development of the other.

I'd put some font emphasis on the "test" related words of the above
extracts, just to be clearer.

>   1.2 Navigating through this document.

I think this section should acknowledge the reuse of the WAI model, so 
that people can actually switch their mind to it, instead of wondering 
what's different but similar.

>   1.3 Terminology
> 
>    The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be
>    used as defined in RFC 2119 [39][RFC2119].

Since the checkpoints define the requirement/normativeness level, and
they use priority instead of verbiage, one may wonder why we say we're 
going to use the RFC2119 model. The WAI guidelines do not refer to it
I think.

>    Checkpoint 1.1.Priority 1.In charters include section for test
>    suites/tools plans. Plan to have at least a commitment to "QA level
>    three".

The WAI style (sorry to always bring it up, but that's the reference
in the area) is to put the Priority level after the checkpoint text,
either in parenth or bracket, usually using a different color
scheme. See /tr/wcag or /tr/uuag.
I like the UAAG approach of just the letter in red and in parenth at
the end of the line: (P1)

>    This 10-point enumeration is derived from a [52]proposal to the QA
>    mail list, after the 4/2001 QA Workshop.

First, it's unclear what relation exists between this commitment scale 
and the requirement expressed in the surrounding checkpoint, so it's
kind of confusing.

Second, somewhow, I think the table is too complex and should be
reorganized more efficiently: half of text is just a duplicate of the
same sentence. David's original linear list was simpler if I remember
correctly.
 
>    Checkpoint 1.2.Priority 2.In charter, specify completion and
>    publication of test materials to be a criterion for CR-exit and
>    PR-entrance.

>    Checkpoint 1.3.Priority 1.In charter, address QA staffing commitments.

I rather reverse the priority of those two: what's more important is
that the WG promises to complete its QA work, not that they identify
QA staff ahead of time.
 

>    The last item requires some elaboration. Depending on the nature of
>    the test, it may actually be sufficient to judge non-compliance of the
>    subject. Based on [54]taxonomy the following is valid:
>      * for conformance testing of the (web) content: statement 1 is
>        false, but statement 2 is true
>      * for content/driven conformance testing of the tools/product: both
>        statements are true
>      * for interfaces (API) conformance testing of the tools or
>        libraries: both statements are true
> 
>      [54] http://www.w3.org/QA/WG/2002/framework-20020118/TAXONOMY

broken link
(should run ,checklink on all documents, Dom will not publish
otherwise)
 
>    Guideline 4. Allocate staff for QA process.
> 
>    Before starting QA development, Working Group has to appoint several
>    key participants of QA process.

not sure a spell checker is going to catch that, but the english is
poor, I suggest:
  Before starting on QA development, a Working Group has to appoint several
  key participants for its QA process.
 
or some such

>    Depending on the plan for origin of the test suite WG can appoint test
>    moderator from

same here, missing articles..

>      * WG members (and other W3C members)
>      * W3C QA staff

in fact, it can come from anywhere, it just has to be part of the WG.
 
>    When test suite is developed outside of Working Group, a moderator
>    usually already exists where test suite coming from. It is recommended
>    to invite moderator to become WG member, since he/she needs access to
>    all WG materials and resources. See also section on [55]transferring
>    test suite from outside W3C.

(english getting really hard to read...)

>    Guideline 6. Produce QA process document.

long guideline, either we'll have to split it or to combine
checkpoints (by experience). not needed for first PD.
 
>    Checkpoint 6.1.Priority 2.Review Technical Guidelines before writing
>    QA Process document and starting test materials development.

not verifiable really, so should not be there
 
>    Taking into account the nature of the future standard, before starting
>    test suite development, test suite implementer (WG or 3rd party)
>    should look at [57]Technical Guidelines and decide what approach to
>    take for test materials organization, tests criteria, etc.
> 
>      [57] http://www.w3.org/QA/WG/2002/framework-20020118/QA-TECH-GUIDE

broken link, maybe should point at DOM process for now
 
>    Checkpoint 6.2.Priority 2.In QA Process document, explain contribution
>    process.

this one looks like a dup of 6.5, or 6.5 + 6.2, in any case, it looks
like it's already covered by following checkpoints
 
>    The following example may serve as a guideline for tests review
>    process

maybe mention that this kind of materials will eventually go into the
example/technical companion

>    Checkpoint 6.7.Priority 3.In QA Process document, describe how vendors
>    can publish test results for their products, if applicable.

this one similar to 3.2 and 3.3, should probably go there or disappear
 
>    The following procedure is recommended:
> 
>    Before transfer WG with the help of QA WG:

same remark as above, example/technique stuff, ok to keep in FPWG but
should mention that it will go away eventually
 
> 3. WG relationship to QA Activity

Is time the only reason why this section is not turned into a
guideline/checkpoint layout ?

Should be mentioned if that's the case and we don't have time to
change it before publication. I think we should take the time though.

>      * The conformance level satisfied: "Level 1", "Level 2", or "Level
>        3".
>    Example:
> 
>    This QA processes and operations of this Working Group [???] conform
>    to W3C's "QA Framework: Process & Operational Guidelines", available
>    at [...tbd...], Level 2.
> 

you mean A, AA, etc

Received on Friday, 25 January 2002 10:40:22 UTC