Re: Updated Test Guidelines draft - 0701

Here are some comments about the 0701 draft Test Guidelines.

1.)  TestGL Glossary:  we should start such a chapter, and throw stuff in 
there which needs to be defined (we can work out the definitions later, and 
migrate at least some to the QA Glossary if appropriate.  Or migrate 
generic versions to QA Glossary, leaving expanded "functional" definitions 
here.  Or ...  But it will make the document more readable during development).

2.) "Levels of Conformance" (1.3, 4.9, other?):  this doesn't fit well with 
SpecGL.  We need to rework it in terms of "conformance variability" 
model.  No suggestion yet.

3.) "The Untestables":

1.4 (discretionary), 1.5 (optional), 1.6 (undefined/ambigous), 1.7 
(explicitly undefined), 1.8 (contradictory)

Actually, they're not all untestable.  The first two are conditionally 
testable, based on choices that the product has made, and the test results 
can be factored into conformance statements about the product.  The last 
three do not lend themselves to any *conformance* testing (although, one 
could write diagnostic tests to see what a product does.)  Perhaps these 
are useful distinctions.  And I can see at least 5 entries for the TGL 
Glossary from these 5 checkpoints.

4.) CK1.3:  It looks like an issue in here, as the dd and WG 
interpretations seem to be opposite: choice from enumerated list versus 
custom definition.

5.) CK2.3:  "sample test scenario" seems like a candidate for further 
definition, explanation, example (or all three).

6.) CK2.4:  reference to "discretionary and vague":  I can see 
"discretionary" here (and "optional"), but I'm not sure about the other 
"untestables".

7.) "The Unverifiables"

4.1, 4.4, 4.5, 5.1, 5.4, 5.6, 5.7:  these are unverifiable or borderline 
unverifiable as stated.  There are a couple of distinct 
problems:  subjective metrics ("easy" in 4.4, "ease of use" in 4.5 and 
5.4); difficult to measure or confirm ("review available..adopt if 
applicable" in 4.1, 5.1); borderline ("suitable to publish" in 5.6, 
"sufficient to investigate" in 5.7).  I think we can and should have a 
checkpoint on each of these topics, but they need some creative 
reworking.  Some, e.g., the last two, could be salvaged by careful 
definition of criteria in the (future) prose explanation of the checkpoint, 
or in the Extech companion.

8.) Candidates for TGL Glossary:  test framework, test case management.

9.) CK1.2, [KG] "need definition of test assertion":  for a functional 
definition for TestGL, I like the one at 
http://lists.w3.org/Archives/Public/www-qa/2002May/0023.html.

10.)  Do we intend to address any goodness principles on test materials? We 
don't seem to go beyond frameworks.   E.g.,  should they (TM) be 
self-documenting?  Should they use W3C technologies and standards, instead 
of proprietary or non-standard technology (e.g., ECMAscript versus private 
scripting language, etc)?

11.)  Finally, CK6.2 raises a question.  In SVG conformance testing, we 
made a design that called for light-weight breadth-first TS ("Basic 
Effectivity") to be written first.  Its tests were simple and not 
necessarily even atomic, but rather provided some light coverage of all of 
the SVG functional areas.  Its value is diagnostic, "pre-conformance" if 
you like -- has the implementation made any effort at all in given 
functional areas?  Its application would typically precede a detailed 
atomic probing of all of the bits of the functional areas.  The overall 
approach is something we called "progressive testing", and the progressive 
referred both to the allocation of resources to build the TS, and the 
application of the TS to an implementation under test.  (In fact, due to 
the hideous cost of building graphics tests, SVG has never gotten beyond 
the ~125 BE tests, to the thousands of DT tests that would make a 
high-confidence conformance suite, and has never adopted a framework like 
XSLT for accepting and integrating DT tests.)

We considered this BE TS to be immensely valuable, but I'm not seeing how 
it or our overall approach would fit into the TestGL.  (Btw, there are 
things in the SVG TS that will fail and ought rightly to fail some 
checkpoints, so I'm not just being a SVG TS chauvinist here.)

All for now,
-Lofton.

Received on Tuesday, 2 July 2002 20:57:48 UTC