RE: Testing Guidelines plan

Hi, David,
I tried to address the issues you brought up in the revised doc but I
haven't answered them directly.

> VAGUENESS (Checkpoints 1.5, 4.6):
Totally agree with your assessment, in the spec we removed the term
vagueness and changed it to union of:
 - unitentionaly undefined
 - ambiguously defined 
We also separated out contradictory behaviors - which seems to be
similar to me...
> Scanning the documents to detect vagueness and cataloging feedback
>sent to the spec editors are two different things; which would this
>checkpoint espouse?
The checkpoint targets both detecting vagueness for next erratas and
cataloging "invalid" tests so that they can be reused later once errata
is issued. 

I've seen your other comment on the intentionally undefined behaviors -
so far we tried to go further and separate 
  - discretionary behaviors (as defined set of choices) and 
  - intentionaly undefined behaviors. (like - specification does not
define any requirements on the limits of the XML tree depth that
processor is able to process).

>TARGET AREAS (Guideline 2):
By test area I understand a set of rules described in the specification
that tester groups together based on some commonality.
Needed mainly for categorization of the tests in the test suite. Usually
the            testing areas match the specification ares/content, but
sometimes it is easier to define them based on some other criteria, like
applicable testing methodology, user scenarios, etc. If there is no 1:1
mapping between test areas and specification areas/content, relationship
between tests and specification should be still traceable via test to
test assertions mapping

>DIMENSIONS OF VARIABILITY (Checkpoints 4.7, 4.8, 4.9):
Agree, so it raises the priority, unless test suite does not cover
corresponding test assertions.

>REPEATABILITY (Checkpoints 4.2, 4.3, 4.10, 5.2, 5.7, 5.8, 7.2):
We should add a checkpoint related to this.

Thanks!

-----Original Message-----
From: David Marston/Cambridge/IBM [mailto:david_marston@us.ibm.com] 
Sent: Wednesday, June 19, 2002 11:10 AM
To: www-qa-wg@w3.org
Subject: Re: Testing Guidelines plan



I don't plan to be in the teleconference tomorrow, so I'll send along
these observations:

VAGUENESS (Checkpoints 1.5, 4.6):
Ideally, the specs have no vagueness. Test developers can identify
the "known" areas of vagueness, but must synchronize with errata.
Scanning the documents to detect vagueness and cataloging feedback
sent to the spec editors are two different things; which would this
checkpoint espouse?

TARGET AREAS (Guideline 2):
Is this like the product classes, profiles, or modules that we have
been discussing in Spec Guidelines? If so, the terminology should
be aligned.

DIMENSIONS OF VARIABILITY (Checkpoints 4.7, 4.8, 4.9):
One could produce a test suite that just tests the bottom level, or
no options, or no extensions, but one must make the test suite
adaptable to the discretionary choices.

REPEATABILITY (Checkpoints 4.2, 4.3, 4.10, 5.2, 5.7, 5.8, 7.2):
Any published results should contain sufficient information about
the test conditions so that an independent lab can set up the
same conditions, run the same suite, and (ideally) get the same
results for the same test subject.
.................David Marston

Received on Tuesday, 2 July 2002 01:19:21 UTC