RE: new draft "Framework: Process & Operational Guidelines"

Thanks, Daniel!
Quick comment on the last question:
>> 3. WG relationship to QA Activity
>Is time the only reason why this section is not turned into a
guideline/checkpoint layout ?
>Should be mentioned if that's the case and we don't have time to change
it before publication. I think we should take the time though.

No, it was intentional. This chapter doesn't require anyone outside of
QA WG to conform to any rules, it rather describes the process QA WG
follows. I found it a bit confusing to write guidelines/checkpoints for
ourselves, hence left this chapter as plain text.
Any arguments why should we change it, except following WAI example?


> -----Original Message-----
> From: Daniel Dardailler [mailto:danield@w3.org] 
> Sent: Friday, January 25, 2002 6:42 AM
> To: Lofton Henderson
> Cc: www-qa-wg@w3.org; ij@w3.org
> Subject: Re: new draft "Framework: Process & Operational Guidelines" 
> 
> 
> 
> 
> >    This document is a Working Draft, made available by the [18]W3C
> >    Quality Assurance (QA) Activity for discussion on the QA email 
> > list.
> 
> a bit picky but it's "for discussion on the QAWG mailing list."
> 
> >    All review comments against the previous version
> >    have been accepted and implemented, except as indicated in an
> >    associated [19]disposition of comments document. This 
> version still
> 
> The disposition of comments actually lists a bunch of "taken" 
> comments, so it's unclear if it is not up-to-date or if the 
> description here is wrong.
> 
> 
> >    This document is part of a family of QA Framework 
> documents designed
> >    to improve the quality of W3C specifications as well as their
> >    implementation by solidifying and extending current 
> quality practices
> >    within the W3C. The QA Framework documents are:
> >      * Introduction
> >      * Specification Guidelines
> >      * Process and Operational Guidelines (this document)
> >      * Technical Guidelines
> 
> I suggest we use the same order as usual
>  
> >    Not only
> >    are the WGs the consumer of these guidelines, they are also key
> 
> consumers
> 
> >    Although not explicitly stated, the W3C Process Document 
> supports the
> >    development of conformance test materials.
> >     1. "[.] groups may produce technical reports, review the work of
> >        other groups, develop sample code or test suites, 
> etc." (sec. 3.)
> >     2. "W3C should make every effort to maintain its Recommendations
> >        (e.g., by tracking errata, providing test bed applications,
> >        helping to create test suites, etc.)" (sec4.)
> >     3. In an effort to meet these suggestions and address the
> >        implementation requirements of the Process Document, 
> some Working
> >        Groups have included the development of conformance 
> materials as
> >        part of their CR-exit and PR-entrance criteria 
> (e.g., SVG, UAAG,
> >        XSL-FO). This makes sense, since it is natural for 
> test suites and
> >        implementations to develop in parallel - each is a 
> help to the
> >        development of the other.
> 
> I'd put some font emphasis on the "test" related words of the 
> above extracts, just to be clearer.
> 
> >   1.2 Navigating through this document.
> 
> I think this section should acknowledge the reuse of the WAI 
> model, so 
> that people can actually switch their mind to it, instead of 
> wondering 
> what's different but similar.
> 
> >   1.3 Terminology
> > 
> >    The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", 
> "SHALL NOT",
> >    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and 
> "OPTIONAL" will be
> >    used as defined in RFC 2119 [39][RFC2119].
> 
> Since the checkpoints define the requirement/normativeness 
> level, and they use priority instead of verbiage, one may 
> wonder why we say we're 
> going to use the RFC2119 model. The WAI guidelines do not 
> refer to it I think.
> 
> >    Checkpoint 1.1.Priority 1.In charters include section for test
> >    suites/tools plans. Plan to have at least a commitment 
> to "QA level
> >    three".
> 
> The WAI style (sorry to always bring it up, but that's the 
> reference in the area) is to put the Priority level after the 
> checkpoint text, either in parenth or bracket, usually using 
> a different color scheme. See /tr/wcag or /tr/uuag. I like 
> the UAAG approach of just the letter in red and in parenth at 
> the end of the line: (P1)
> 
> >    This 10-point enumeration is derived from a [52]proposal 
> to the QA
> >    mail list, after the 4/2001 QA Workshop.
> 
> First, it's unclear what relation exists between this 
> commitment scale 
> and the requirement expressed in the surrounding checkpoint, 
> so it's kind of confusing.
> 
> Second, somewhow, I think the table is too complex and should 
> be reorganized more efficiently: half of text is just a 
> duplicate of the same sentence. David's original linear list 
> was simpler if I remember correctly.
>  
> >    Checkpoint 1.2.Priority 2.In charter, specify completion and
> >    publication of test materials to be a criterion for CR-exit and
> >    PR-entrance.
> 
> >    Checkpoint 1.3.Priority 1.In charter, address QA staffing 
> > commitments.
> 
> I rather reverse the priority of those two: what's more 
> important is that the WG promises to complete its QA work, 
> not that they identify QA staff ahead of time.
>  
> 
> >    The last item requires some elaboration. Depending on 
> the nature of
> >    the test, it may actually be sufficient to judge 
> non-compliance of the
> >    subject. Based on [54]taxonomy the following is valid:
> >      * for conformance testing of the (web) content: statement 1 is
> >        false, but statement 2 is true
> >      * for content/driven conformance testing of the 
> tools/product: both
> >        statements are true
> >      * for interfaces (API) conformance testing of the tools or
> >        libraries: both statements are true
> > 
> >      [54] http://www.w3.org/QA/WG/2002/framework-20020118/TAXONOMY
> 
> broken link
> (should run ,checklink on all documents, Dom will not publish
> otherwise)
>  
> >    Guideline 4. Allocate staff for QA process.
> > 
> >    Before starting QA development, Working Group has to 
> appoint several
> >    key participants of QA process.
> 
> not sure a spell checker is going to catch that, but the 
> english is poor, I suggest:
>   Before starting on QA development, a Working Group has to 
> appoint several
>   key participants for its QA process.
>  
> or some such
> 
> >    Depending on the plan for origin of the test suite WG 
> can appoint test
> >    moderator from
> 
> same here, missing articles..
> 
> >      * WG members (and other W3C members)
> >      * W3C QA staff
> 
> in fact, it can come from anywhere, it just has to be part of the WG.
>  
> >    When test suite is developed outside of Working Group, a 
> moderator
> >    usually already exists where test suite coming from. It 
> is recommended
> >    to invite moderator to become WG member, since he/she 
> needs access to
> >    all WG materials and resources. See also section on 
> [55]transferring
> >    test suite from outside W3C.
> 
> (english getting really hard to read...)
> 
> >    Guideline 6. Produce QA process document.
> 
> long guideline, either we'll have to split it or to combine 
> checkpoints (by experience). not needed for first PD.
>  
> >    Checkpoint 6.1.Priority 2.Review Technical Guidelines 
> before writing
> >    QA Process document and starting test materials development.
> 
> not verifiable really, so should not be there
>  
> >    Taking into account the nature of the future standard, 
> before starting
> >    test suite development, test suite implementer (WG or 3rd party)
> >    should look at [57]Technical Guidelines and decide what 
> approach to
> >    take for test materials organization, tests criteria, etc.
> > 
> >      [57] 
> > http://www.w3.org/QA/WG/2002/framework-20020118/QA-TECH-GUIDE
> 
> broken link, maybe should point at DOM process for now
>  
> >    Checkpoint 6.2.Priority 2.In QA Process document, 
> explain contribution
> >    process.
> 
> this one looks like a dup of 6.5, or 6.5 + 6.2, in any case, 
> it looks like it's already covered by following checkpoints
>  
> >    The following example may serve as a guideline for tests review
> >    process
> 
> maybe mention that this kind of materials will eventually go 
> into the example/technical companion
> 
> >    Checkpoint 6.7.Priority 3.In QA Process document, 
> describe how vendors
> >    can publish test results for their products, if applicable.
> 
> this one similar to 3.2 and 3.3, should probably go there or disappear
>  
> >    The following procedure is recommended:
> > 
> >    Before transfer WG with the help of QA WG:
> 
> same remark as above, example/technique stuff, ok to keep in 
> FPWG but should mention that it will go away eventually
>  
> > 3. WG relationship to QA Activity
> 
> Is time the only reason why this section is not turned into a 
> guideline/checkpoint layout ?
> 
> Should be mentioned if that's the case and we don't have time 
> to change it before publication. I think we should take the 
> time though.
> 
> >      * The conformance level satisfied: "Level 1", "Level 
> 2", or "Level
> >        3".
> >    Example:
> > 
> >    This QA processes and operations of this Working Group 
> [???] conform
> >    to W3C's "QA Framework: Process & Operational 
> Guidelines", available
> >    at [...tbd...], Level 2.
> > 
> 
> you mean A, AA, etc
>  
> 
>

Received on Friday, 25 January 2002 13:14:35 UTC