RE: Observations about TestGL organization

Re-reading the proposals from David and from Patrick&Peter, I'm asking
the question again - should there be anything about process in the
TestGL?

Both proposals as well as the latest draft still leave the process of
the test case development in the TestGL, even though part of it is
covered in the OpsGL. 

I think this will continue generate concerns from QA community, saying
that we mix "What is a good test suite?" and "How to build a good test
suite?".

TestGL was going slowly to having less and less of the process in it. 

Should we just move the operational checkpoints from the old Gd4, Gd6
and probably Gd7 to the Ops GL?
Thanks





-----Original Message-----
From: David Marston/Cambridge/IBM [mailto:david_marston@us.ibm.com] 
Sent: Saturday, March 08, 2003 1:45 PM
To: www-qa@w3.org
Subject: Observations about TestGL organization






This message summarizes my overall view of the Test Guideline (GL) and
Checkpoint (CP) organization. I hope to send along some more detailed
ideas later.

The CPs have to be flexible because different WGs can approach the
whole testing strategy in different ways, and they may choose different
levels of commitment. The Commitment Table at [1] has more levels than
checkpoints, which becomes useful in this discussion of TestGL. In
other words, if a WG commits to a QA level that is between two OpsGL
checkpoints, it still helps them meet more TestGL CPs. Notably, test
cases may be written by the WG or contributed, or both, which ties to
the difference between QA levels 3 and 4.

Without intending to, I devised a new allocation of the CPs across 7
new guidelines. Like the SpecGL guidelines, I have tried to arrange
them in a logical progression.

1. The current GL 7 could be split, with one part being a new GL 1 about
planning. It would include old 7.1, 1.1, a revised 1.2 allowing test
assertions (TAs) to be explicit or not, something about test case
contributions, and maybe old 3.2.

Upon fulfilling the above guideline, the WG stands ready to divide the
test materials work among planning the test-case management as static
objects, planning the methods of running and reporting, generating a
list of cases needed, and writing test cases.

2. The next logical GL is probably old GL 1. CPs 1.3 through 1.8 cover
the Dimensions of Variability (DoV) and belong here, as does old
1.10. Old 1.9 belongs here too, but should be broadened to cover
test planning and test case development as ways of checking the spec,
especially in pre-CR stages.

3. The next GL is old GL 6, since it's part of the planning and
analysis phase. CPs 6.3, 2.2, and 4.14 all deal with resolving the
match between tests and spec, whether through the intermediary of
TAs or not, and should be refactored in any new arrangement of CPs.

4. Then comes whatever is left of GL 3. Old CP 2.1 probably goes here
as well. GL 2 is thus refactored away.

Up to this point in the progression, the WG has identified "what
tests ought to be written", though whether they attempted to make
an exhaustive list depends on their committed QA level.

5. Next comes that portion of GL 4 that deals with collecting test
cases and managing their role and status. All the DoV should be
addressed again, as shown by old CPs 4.6 to 4.10 (that parallel old
1.3 to 1.8). An interesting new flag comes from old 1.4: tests can
be flagged as conformance tests vs. interoperability tests, which
allows useful contributions to be retained in a lesser status
rather than rejected outright. Old CPs 4.12 and 4.4/4.13 (nearly
the same?) also belong here.

Up to this point in the progression, the WG has the plan and the
methods for gathering and storing test cases. Notice that CPs
(old) 1.4, 4.3, and 4.12 can be applied on an individual test
case/assertion basis.

6. The next GL should be that portion of old GL 4 that deals with
running tests as opposed to writing them. Old CPs 4.2, 4.3, and
4.5 go here. I'm not sure if CP 4.1 goes anywhere, but it could go
here.

7. The final GL takes old GL 5 and sharpens it a bit. All the CPs
5.1 to 5.5 stay here, plus 4.11 and 7.2.

This plan is very rough! I expect that CPs will change as they fit
into this new progression.
.................David Marston

[1] http://www.w3.org/TR/qaframe-ops/#commitment-table

Received on Monday, 10 March 2003 01:37:22 UTC