W3C home > Mailing lists > Public > www-qa@w3.org > April 2003

OpsGL QA-commitment-group

From: Lofton Henderson <lofton@rockynet.com>
Date: Tue, 29 Apr 2003 16:40:05 -0600
Message-Id: <5.1.0.14.2.20030429150606.031c35e0@terminal.rockynet.com>
To: www-qa@w3.org

This is probably the most significant of the OpsGL issues, and is the only 
real issue group where we had lots of comments.  For email discussion, and 
the agenda of the next OpsGL telecon.

Ref:  http://www.w3.org/QA/WG/lc-issues

Last Call Issues:  3, 60.2, 72.2, 72.3, 83, 107

Following are:  Synopsis, comment excerpts, analysis, and finally proposal.

Synopsis:
=====
The QA commitment table in GL1, and the checkpoints CP1.1, 1.2, 1.3, 
apparently set up an alternative (7-level) conformance scale for working 
groups, that interacts with and in places contradicts the checkpoints and 
conformance levels in SpecGL and in OpsGL itself.

The comments:
=====
LC-3:  "Remove the Commitment Table and CP1.1, 1.2, 1.3. These items do not 
add information and are redundant. To satisfy Level 3 (5 or 7), you must 
satisfy other checkpoints in the OpsGL. In fact, if we did our jobs right, 
Level 3 should equal Conformance Level A (satifying all P1), Level 5 should 
equal Level Double A, and Level 7 = Triple A. Isn't it a goal to get WGs to 
conform to the OpsGL? That would mean that they must satisfy all the P1 
checkpoints. Thus, they must have a commitment level to P1. So, why do we 
use a new term - Level 3."

LC-60.2:  "...checkpoints 1.1 - 1.3 [are] very confusing [...] "compound 
checkpoints" that incorporate multiple other checkpoints. We don't use this 
structure anywhere else in our docs - why here? The document states "This 
seven-point enumeration is derived from the proposal to the QA mail list, 
after the 4/2001 QA Workshop", but how we got here is not really 
interesting to the reader.  The 'sub-checkpoints' on the 'left hand side' 
of the table are all spec-related. Either these duplicate checkpoints from 
SpecGL, or they should be incorporated into that document. Those on the 
'right hand side' are operations-related, but they seem to overlap with 
other checkpoints specified in this document.  Recommendation: drop the 
compound structure, make sure that all the spec-related sub-checkpoints are 
covered by SpecGL, and move the 'test materials' sub-checkpoints into the 
body of this doc."

LC-72.2:  "The QA level system is confusing. At least three different 
metrics are used: priorities of checkpoints (P1, P2, P3), degrees of 
conformance (A, AA, AAA) and QA levels (1..7). The level systems seems 
unnecessary and can be covered with priorities and 
degrees.  Proposal:  Drop QA level system and use priorities instead."

LC-72.3:  "...[CP1.1] is of P1 and requires to commit to QA level three 
which in its turn refers to checkpoints of P1, P2, P3 priority."

LC-83:  "All specs include seven "levels" of conformance. However, only 
three levels are actually public; these are called "priorities".  Proposal: 
Collapse the seven levels into the three real ones (now called priorities), 
since these are the basis of conformance measurement."

LC-107:  "The introduction of A[A][A]-conforming as a synonym for the three 
priorities is rather absurd, as is the creation of four levels that will 
never be used. Proposal: Pick one set of terms and toss the remainder. "

Analysis
=====
The commitment table, its source materials, and its checkpoints date back 
around 16-18 months.  The GL family -- SpecGL, OpsGL, and now TestGL -- 
have developed in the meantime and overtaken these GL1 bits of 
OpsGL.  Inconsistencies have been introduced, such as OpsGL CP1.1 (priority 
1) requiring implementation of other OpsGL checkpoints of p2 and p3, and 
implementation of SpecGL checkpoints of p2 and p3.

While of historical interest, almost all of what is accomplished here can 
be done more cleanly and consistently by reference to our uniform GL 
conformance model.  There is one bit that is not actually covered by 
reference to the other GL documents.  The commitment table requires the 
commitment to provide test materials, and this is not required anywhere 
else in the GL documents.  Therefore in the proposal below, I have 
distilled the "produce or adopt" TM requirement, from the right-hand column 
of the table, that pertains at table level 3, 5, and 7, and have added as a 
bullet in (respectively) CP1.1, 1.2, and 1.3.

Remember, these checkpoints 1.1, 1.2, 1.3 are about the WG making and 
documenting an a' priori commitment to do something (in Charter or asap 
thereafter), rather than about WG's measured performance on something.  It 
is for this reason -- advance commitment -- that I think the CPs can't go 
away altogether.  I.e., it is a different issue to eliminate the CPs versus 
fixing them, and that issue would be:  should the WG define and its 
intentions and commitment early, or not?

Proposal
=====
Replace Checkpoints 1.1, 1.2, 1.3 with three new checkpoints.

CP1.1:  Commit to at least uniform degree A conformance.  [Priority 1]
ConfReq:  the WG MUST commit that it will:
	* attain degree A conformance to OpsGL
	* attain degree A conformance to SpecGL in any Recommendations it produces;
	* produce or adopt at least some test materials for each of the WG's 
specifications before it becomes Recommendation;
	* attain degree A conformance to TestGL in any Test Materials it produces 
of adopts;
[...additional blither about new versus existing working groups...]

CP1.2:  Commit to at least uniform degree AA conformance.  [Priority 2]
ConfReq:  the WG MUST commit that it will:
	* attain degree AA conformance to OpsGL
	* attain degree AA conformance to SpecGL in any Recommendations it produces;
	* produce or adopt at least some test materials for each of the WG's 
specifications before it becomes Recommendation;
	* attain degree AA conformance to TestGL in any Test Materials it produces;
[...additional blither about new versus existing working groups...]

CP1.3:  Commit to at least uniform degree AAA conformance.  [Priority 3]
ConfReq:  the WG MUST commit that it will:
	* attain degree AAA conformance to OpsGL
	* attain degree AAA conformance to SpecGL in any Recommendations it produces;
	* produce or adopt complete test materials before Recommendation, where 
complete is defined as: at least one test case for every identifiable 
conformance requirement of the specification.
	* attain degree AAA conformance to TestGL in any Test Materials it produces;
[...additional blither about new versus existing working groups...]

### end ###

Regards,
-Lofton.
Received on Tuesday, 29 April 2003 18:38:42 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Sunday, 6 December 2009 12:13:59 GMT