Some TestGL comments

I am about half finished with a careful reading of TestGL [1].  Here are 
some comments.  I'll try to finish up in the next couple days.

Caveat.  Although I have tried to get a "big view" by looking at the 
overall shape of guidelines/checkpoints -- the checklist [2] is good for 
this -- no conclusions or proposals have jumped out at me yet.  Maybe 
that's a good sign.  The first 3 or so GLs seem pretty well composed, at 
this outline level.

Minor Editorial.  I have lots of small editorial comments -- punctuation, 
spelling, grammar, etc.  I have marked them on a paper copy and will give 
them to editor, Peter.

Substantive and Major Editorial:

"Status".  We say that we have not addressed priority levels.  Do we need 
to do this by next draft, the 2ndPWD?  Or can it wait?  I 
suggest:  2ndPWD.  Reason.  TestGL is lagging pretty far behind SpecGL and 
OpsGL, and we ought to close the gap a little.

Ch.1 (Introduction):  This needs to be rewritten to conform to SpecGL 
requirements:  at least, clearly define Scope, class of product, audience, 
etc.  Proposal.  Start with sub-section skeleton from OpsGL/SpecGL.  A lot 
of the sub-section text will be directly usable.  Re-distribute the various 
bits in the current TestGL introduction, and add things that are missing 
(like Scope definition).

Ch.1, pgph.3:  "This document aims at giving guidelines for testing 
implementation's conformance with the W3C specifications."  Is this really 
within the scope of TestGL, "guidelines for testing"?  It is my view that 
the scope is a set of guidelines about the nature and contents of the test 
materials themselves.

Ch.1, pgph.4/bullets/pgph.5:  the purposes of these bits are unclear at 
first.  Proposed revision -- replace it all with, "The main goal of the 
checkpoints in this document is to verify that a test suite provides 
sufficient information to answer these questions:", followed by the bullet 
list.

Ch.1, last pgph:  change "semantic requirements" to "test assertions" (they 
are synonymous in the QA Glossary, and TA is our preferred usage.)

Sec1.1, bullets & following paragraph:  this is very difficult to 
follow.  In fact, I'm not quite sure what we're getting at in the 
paragraph.  Rewrite/clarify.

Sec1.1, 2nd bullet list:  change "compliance" to "conformance", here and 
throughout the document.  Similarly, "compliant" to "conformant", "comply" 
to "conform", etc. (According to QA Glossary, they are synonymous, and 
"conformance" is our preferred usage.)

Sec1.1, 2nd bullet list, 2nd bullet:  Is this true?  Does a test suite 
"provide conformance criteria for implementations"?  I think "no".  It is 
the specification that provides the conformance criteria, isn't it?  (A 
test suite measures an implementation against conformance criteria.)

Sec1.2, 1st sentence:  "The Guidelines of this document follow the 
structure of the test suite quality criteria outlined above".  Clarify this 
reference, as I'm not quite sure what it is pointing to above.

Sec1.5, Definitions:  SpecGL has this section near the end.  We should be 
consistent.

Sec1.5, "test area":  It would be nice if the definition said more.  More 
explanation of the meaning, and/or include a "for example".

All Checkpoints:  All checkpoints need to be revised in the style of SpecGL 
and OpsGL.  For 2ndPWD, minimally we need:  Statement of the checkpoint, 
followed by the tagged (class="TA") section starting with "Conformance 
requirements:".  This is going to be more than a formatting exercise.  I 
think it will uncover some issues when we try to do it, because we have to 
state some verifiable requirements.  A simple example of the problem 
is:  in all of the checkpoints that start, "Identify..", "Declare..", etc, 
an interesting question is "where?".  In OpsGL, we had to say "in the 
charter", or "in the QAPD", or "in some minuted consensus document".

All Checkpoints:  In addition to refining the structure to CP-statement 
plus Conformance-requirements, SpecGL and OpsGL have the subsequent 
verbiage structured into "Rationale..." and "Discussion..." (either one may 
be missing, depending on the CP).  This waited till Last Call for OpsGL, so 
maybe TestGL can postpone this for one more cycle.  However, coming up with 
a convincing "Rationale" is a good exercise.

CP1.3, "etc" and "e.g.":  Are level, profiles, and modules exhaustive 
here?    Or are there others?  If exhaustive, then eliminate "etc" and 
"e.g.".  If not, then list some more, or give a qualitative explanation of 
what sort of other things might be on the list.

CP2.1:  I think that the bullet list should be in the verbiage of GL2, not 
in EX-TECH.  (Or in both places.)

CP2.2, 1st sentence:  "test areas" needs more explanation.

CP2.2, 2nd [EX-TECH]:  The concept of sample testing scenarios needs better 
explanation.  I'm unclear what it is.  Consider:  "sample testing 
scenarios...these are not actual tests, but rather test examples".  The 
last parts tempt me to think that they are something like a "prototype test 
case".

GL3 1st pgph:  "How does the test suite verify conformance to the 
specification?"  Answer:  it doesn't!  (There is a problem with the statement.)

CP3.1:  "test area".  Define and explain this in the verbiage of the 
Guideline 3.

CP3.2:  Multi-sentence statement of the checkpoint.  Simplify the statement 
of the checkpoint, and put the details into the "Conformance 
requirements".  Example.  "Identify testing techniques used."  "Conformance 
requirements: the test materials ??? document MUST enumerate applicable 
publicly available testing techniques, and identify those that have been 
used in the test materials."

CP4.1 through 4.14, 5.2, 6.2, 6.3.  Same comment as CP3.2.

CP5.3, 7.2, and maybe others:  Although not multi-sentence, simplification 
and partitioning comments of CP3.2 apply.

All for now,
-Lofton.

[1] http://www.w3.org/TR/2002/WD-qaframe-test-20021220/
[2] http://www.w3.org/TR/2002/WD-qaframe-test-20021220/qaframe-test-checklist

Received on Friday, 28 February 2003 19:32:06 UTC