TestGL feedback

Here are my comments about the 23 April draft of TestGL:

Ckp 1.1 says:
"Note, a WG may have multiple test suites for different parts of a
specification (e.g., profiles or modules) or for different applications
of the specification (e.g., bindings, things like CSS in MathML CSS
with WCAG etc)"

I prefer modularization of one suite to accommodate all DoV, with the
possible exception of class of product. We should discourage piecemeal
conformance claims. Actually, this whole observation about multiple
or modular suites fits better under 1.2, structure of TM. (Also note
that 3.2 is expected to address DoV.)

Regarding Ckp 1.2 (and 2.1):
If the spec A-conforms to SpecGL, the test writers should be able to
determine DoV readily. Tests can be "categorized" by module, level,
involvement of discretionary items, etc., which were enumerated or
pointed to by the Conformance Section. Being able to flag a test for
a deprecated feature brings up another form of modularizing the
suite which is *not* a DoV in the spec: versions. A case can be used
in normal status when the suite is filtered for version 1.x, still
used but in deprecated status when the suite is filtered for version
2.x, and left out when the suite is filtered for version 3.x and up.

I generally agree with Sandra Martinez' point about stating the point
of a test, and I think it should be addressed under GL 3 somewhere.

Under GL 4, suite development:
I'd like to see a checkpoint about gathering and reviewing test cases.
You could use OASIS guidelines for test case review as a model. And
maybe in TestET, we would say something about tying test data from
the Use Cases or Requirements documents into real conformance tests.

Regarding Ckp 5.2:
Why is that "not" in last sentence?

Regarding Ckp 5.4:
Rather vague. Is there some reason why we don't just say each test case
must have a unique name? That allows easy discussion of cases in email
and teleconferences, in addition to providing the "association" that
the Ckp calls for. And why is there no mention of storage of results?

Regarding Ckp 6.2:
Say "must" rather than "should" in the Ckp, per previous QAWG action
on this issue for other QAWG checkpoint documents.

Regarding Ckp 6.4:
As I read this, it seems like the harder and riskier way to do it. Why
not just filter the tests to be run, run only applicable ones, then
report results on all you ran? We want test labs to run and report on
all applicable tests as the default mode of operation. Leave the
selective reporting to vendors who need to hide behind uncertainty
because their products are inferior.

Finally, down in section 5 (Acknowledgments), please change my
corporate affiliation to "IBM Research".
.................David Marston

Received on Monday, 5 May 2003 23:00:40 UTC