RE: Updated Test Guidelines draft - 0701

One clarification...

At 06:24 PM 7/2/02 -0700, Kirill Gavrylyuk wrote:
>[...]
>10) I see Framework checkpoint as requirements. "Goodness principles" is
>the solution that you can use to satisfy the requirement. We will have
>them, most likey in the Tech Ex document. Your 2 examples are addressed
>by requirements in Ck 4.11, 4.2, 4.3.

4.2/4.3 and "use W3C or other standard technologies" are related, but not 
identical.  It would be possible to make the framework available on all 
platforms without using standard technologies.


>11) This is exactly the reason for the Ck 2.3, 2.4. I call them "sample
>tests". We definitely need to work on the prose more,

...and maybe terminology -- 2.3/2.4 don't say "sample tests', but rather 
"sample test scenarios", which is not very suggestive.  Furthermore, 2.3 
starts off "Before creating test cases for...," and ""Those are not actual 
tests..."  The SVG Basic Effectivity TS -- a functionally comprehensive 
sample test suite of 125 actual tests -- was a useful deliverable in its 
own right.  Its 125 test cases were "samples" in the sense that 
collectively they comprised a uniform (but somewhat sparse) sampling of the 
entire functional breadth of SVG.

Long story short, I'm still a bit little uncertain whether we're talking 
about the same thing.  But I'll take your word for it and look out for the 
next revision.

-Lofton.



>-----Original Message-----
>From: Lofton Henderson [mailto:lofton@rockynet.com]
>Sent: Tuesday, July 02, 2002 6:00 PM
>To: www-qa-wg@w3.org
>Subject: Re: Updated Test Guidelines draft - 0701
>
>
>Here are some comments about the 0701 draft Test Guidelines.
>
>1.)  TestGL Glossary:  we should start such a chapter, and throw stuff
>in
>there which needs to be defined (we can work out the definitions later,
>and
>migrate at least some to the QA Glossary if appropriate.  Or migrate
>generic versions to QA Glossary, leaving expanded "functional"
>definitions
>here.  Or ...  But it will make the document more readable during
>development).
>
>2.) "Levels of Conformance" (1.3, 4.9, other?):  this doesn't fit well
>with
>SpecGL.  We need to rework it in terms of "conformance variability"
>model.  No suggestion yet.
>
>3.) "The Untestables":
>
>1.4 (discretionary), 1.5 (optional), 1.6 (undefined/ambigous), 1.7
>(explicitly undefined), 1.8 (contradictory)
>
>Actually, they're not all untestable.  The first two are conditionally
>testable, based on choices that the product has made, and the test
>results
>can be factored into conformance statements about the product.  The last
>
>three do not lend themselves to any *conformance* testing (although, one
>
>could write diagnostic tests to see what a product does.)  Perhaps these
>
>are useful distinctions.  And I can see at least 5 entries for the TGL
>Glossary from these 5 checkpoints.
>
>4.) CK1.3:  It looks like an issue in here, as the dd and WG
>interpretations seem to be opposite: choice from enumerated list versus
>custom definition.
>
>5.) CK2.3:  "sample test scenario" seems like a candidate for further
>definition, explanation, example (or all three).
>
>6.) CK2.4:  reference to "discretionary and vague":  I can see
>"discretionary" here (and "optional"), but I'm not sure about the other
>"untestables".
>
>7.) "The Unverifiables"
>
>4.1, 4.4, 4.5, 5.1, 5.4, 5.6, 5.7:  these are unverifiable or borderline
>
>unverifiable as stated.  There are a couple of distinct
>problems:  subjective metrics ("easy" in 4.4, "ease of use" in 4.5 and
>5.4); difficult to measure or confirm ("review available..adopt if
>applicable" in 4.1, 5.1); borderline ("suitable to publish" in 5.6,
>"sufficient to investigate" in 5.7).  I think we can and should have a
>checkpoint on each of these topics, but they need some creative
>reworking.  Some, e.g., the last two, could be salvaged by careful
>definition of criteria in the (future) prose explanation of the
>checkpoint,
>or in the Extech companion.
>
>8.) Candidates for TGL Glossary:  test framework, test case management.
>
>9.) CK1.2, [KG] "need definition of test assertion":  for a functional
>definition for TestGL, I like the one at
>http://lists.w3.org/Archives/Public/www-qa/2002May/0023.html.
>
>10.)  Do we intend to address any goodness principles on test materials?
>We
>don't seem to go beyond frameworks.   E.g.,  should they (TM) be
>self-documenting?  Should they use W3C technologies and standards,
>instead
>of proprietary or non-standard technology (e.g., ECMAscript versus
>private
>scripting language, etc)?
>
>11.)  Finally, CK6.2 raises a question.  In SVG conformance testing, we
>made a design that called for light-weight breadth-first TS ("Basic
>Effectivity") to be written first.  Its tests were simple and not
>necessarily even atomic, but rather provided some light coverage of all
>of
>the SVG functional areas.  Its value is diagnostic, "pre-conformance" if
>
>you like -- has the implementation made any effort at all in given
>functional areas?  Its application would typically precede a detailed
>atomic probing of all of the bits of the functional areas.  The overall
>approach is something we called "progressive testing", and the
>progressive
>referred both to the allocation of resources to build the TS, and the
>application of the TS to an implementation under test.  (In fact, due to
>
>the hideous cost of building graphics tests, SVG has never gotten beyond
>
>the ~125 BE tests, to the thousands of DT tests that would make a
>high-confidence conformance suite, and has never adopted a framework
>like
>XSLT for accepting and integrating DT tests.)
>
>We considered this BE TS to be immensely valuable, but I'm not seeing
>how
>it or our overall approach would fit into the TestGL.  (Btw, there are
>things in the SVG TS that will fail and ought rightly to fail some
>checkpoints, so I'm not just being a SVG TS chauvinist here.)
>
>All for now,
>-Lofton.

Received on Wednesday, 3 July 2002 17:05:59 UTC