- From: David Marston/Cambridge/IBM <david_marston@us.ibm.com>
- Date: Sat, 8 Mar 2003 16:44:35 -0500
- To: www-qa@w3.org
This message summarizes my overall view of the Test Guideline (GL) and Checkpoint (CP) organization. I hope to send along some more detailed ideas later. The CPs have to be flexible because different WGs can approach the whole testing strategy in different ways, and they may choose different levels of commitment. The Commitment Table at [1] has more levels than checkpoints, which becomes useful in this discussion of TestGL. In other words, if a WG commits to a QA level that is between two OpsGL checkpoints, it still helps them meet more TestGL CPs. Notably, test cases may be written by the WG or contributed, or both, which ties to the difference between QA levels 3 and 4. Without intending to, I devised a new allocation of the CPs across 7 new guidelines. Like the SpecGL guidelines, I have tried to arrange them in a logical progression. 1. The current GL 7 could be split, with one part being a new GL 1 about planning. It would include old 7.1, 1.1, a revised 1.2 allowing test assertions (TAs) to be explicit or not, something about test case contributions, and maybe old 3.2. Upon fulfilling the above guideline, the WG stands ready to divide the test materials work among planning the test-case management as static objects, planning the methods of running and reporting, generating a list of cases needed, and writing test cases. 2. The next logical GL is probably old GL 1. CPs 1.3 through 1.8 cover the Dimensions of Variability (DoV) and belong here, as does old 1.10. Old 1.9 belongs here too, but should be broadened to cover test planning and test case development as ways of checking the spec, especially in pre-CR stages. 3. The next GL is old GL 6, since it's part of the planning and analysis phase. CPs 6.3, 2.2, and 4.14 all deal with resolving the match between tests and spec, whether through the intermediary of TAs or not, and should be refactored in any new arrangement of CPs. 4. Then comes whatever is left of GL 3. Old CP 2.1 probably goes here as well. GL 2 is thus refactored away. Up to this point in the progression, the WG has identified "what tests ought to be written", though whether they attempted to make an exhaustive list depends on their committed QA level. 5. Next comes that portion of GL 4 that deals with collecting test cases and managing their role and status. All the DoV should be addressed again, as shown by old CPs 4.6 to 4.10 (that parallel old 1.3 to 1.8). An interesting new flag comes from old 1.4: tests can be flagged as conformance tests vs. interoperability tests, which allows useful contributions to be retained in a lesser status rather than rejected outright. Old CPs 4.12 and 4.4/4.13 (nearly the same?) also belong here. Up to this point in the progression, the WG has the plan and the methods for gathering and storing test cases. Notice that CPs (old) 1.4, 4.3, and 4.12 can be applied on an individual test case/assertion basis. 6. The next GL should be that portion of old GL 4 that deals with running tests as opposed to writing them. Old CPs 4.2, 4.3, and 4.5 go here. I'm not sure if CP 4.1 goes anywhere, but it could go here. 7. The final GL takes old GL 5 and sharpens it a bit. All the CPs 5.1 to 5.5 stay here, plus 4.11 and 7.2. This plan is very rough! I expect that CPs will change as they fit into this new progression. .................David Marston [1] http://www.w3.org/TR/qaframe-ops/#commitment-table
Received on Saturday, 8 March 2003 16:45:07 UTC