- From: pfawcett <pfawcett@speakeasy.org>
- Date: Thu, 20 Mar 2003 16:11:57 -0800
- To: www-qa-wg@w3.org
Ok, here is a stab at trying to map the old Test GL contents to the new Test GL outline. I've sort of done it in two parts, the first part is a simple mapping with out any comments but it wont make much sense unless your looking at the actual TestGL document and the new outline. The second part has the actual checkpoint titles which makes it easier but it gets hard to keep perspective of the 'big picture'. The two together do an ok job of giving the over all idea. Most of what was there before is still there, there are only a few checkpoints that I could not find any mapping for. Many checkpoints are merged into one in the new document (specifically GL1 CP2-9 and GL4 CP4,CP6,CP7-10,CP12 and CP14). Patrick if you think I missed anything your welcome to add stuff. If nothing else this is a good basis for discussion on monday. Peter The summary is: Old Outline New Outline -------------------------------------- GL1 CP1 -> Stays (GL1 CP1). GL1 CP2-9 -> GL2 CP1 GL1 CP10 -> Stays (GL1 CP4). GL2 CP1-2 -> GL1 CP2 and GL3 CP2 GL3 CP1 -> GL1 CP3 GL3 CP2 -> Moved to extec and/or descriptive language. GL4 CP1 -> Moved to extec and/or descriptive language. GL4 CP2 -> Moved to extec and/or descriptive language. GL4 CP3 -> Combined with GL6 CP2. GL4 CP4 -> Part of GL5 CP3 GL4 CP5 -> Part of GL5 CP1 GL4 CP6 -> Related to GL5 CP3 and also OpsGL GL4 CP7-10 -> Combined into G5 CP3 GL4 CP11 -> Not sure if this makes is anywhere.... GL4 CP12 -> Combined into G5 CP3 GL4 CP13 -> Merged into GL3 CP1 GL4 CP14 -> Combined into G5 CP3 GL5 CP1 -> Stays. GL5 CP2 -> Stays. GL5 CP3 -> Stays. GL5 CP4 -> Stays. GL5 CP5 -> Stays but is combined with CP2. GL6 CP1 -> GL2 CP2 GL6 CP2 -> GL4 CP1 GL6 CP3 -> Removed, may already be covered by OpsGL. GL7 -> Stays the same. More in depth analysis: Old G1: Provide analysis of the specification(s). New G1: High level analysis of specification. CP1: Identify the target set of specifications that are being tested. - CP1 is slightly reformed but is still here. It is now included as part of the scope of the test suite (likely still checkpoint 1 too). CP2: Identify test assertions from the target set of specifications. CP3: Group test assertions by levels, profiles, modules, etc. CP4: Identify those test assertions that are part of the conformance criteria CP5: Identify those test assertions that are part of the conformance criteria CP6: Identify optional behaviors in the specification CP7: Identify behaviors that are undefined or defined ambiguously in the specification CP8: Identify explicitly undefined behaviors in the specification CP9: Identify contradictory behaviors in the specification - CP2-9 combine to become the first checkpoint of the new GL2. CP10: List the user scenarios for the specification - CP10 stays here. Old G2: Declare the structure of the test suite. New G2: Low level analysis of specification. CP1: Document the structure for the test suite. CP2: Provide mapping between the test suite structure and the specification structure. - CP1-2 are combined into the CP2 of GL1. Part of CP2 is also now in the new GL3, CP2 of the outline, in that a mapping of test case to assert to normative language is one of the pieces of metadata that system should support. Old G3: Document the testing methodology. New G3: Provide A Test Management System CP1: For each test area, identify the testing approach - CP1 This is now GL1 CP2 or 3. What is being tested, how much, how, what priority. CP2: Identify publicly available testing techniques. List the publicly available testing techniques that have been reused - CP2 is moved to extec or descriptive language Old G4: Provide the test automation and framework New G4: Test Suite Development CP1: List available test frameworks and applicable automation. Identify available test frameworks used. If none, justify why new frameworks are needed and existing ones could not be used. - CP1 is moved to extec or descriptive language CP2: Ensure the framework and automation are platform independent. Demonstrate on 3 platforms. Ensure that the framework and automation are built using open standards. - CP2 is moved to extec or descriptive language CP3: Ensure the framework and automation are applicable to any product or content that implements the specification. Demonstrate with three products or contents. Ensure that the framework and automation are built using open standards. - CP3 is still here but focus is on prototyping and ensuring that the framework is suitable for it's intended purpose. CP4: Ensure the framework makes it easy to add tests for any of the specification areas. Demonstrate, through an example, how tests are added to a specification area. - specification area is one of the meta data criteria that the test management system should support, so this get's combined into GL5 CP3 CP5: Ensure the ease of use for the test automation. Document how the test automation is used. - Part of GL5 CP1 CP6: Ensure the framework allows for specification versioning and errata levels. Explain how specification versioning and errata levels are accommodated by the test framework - Not here except as one of the meta data criteria. Explicit versioning and eratta guidelines also exist in OpsGL. CP7: Ensure the framework accounts for choices allowed by discretionary choices in the specification. Explain how discretionary behaviors are accommodated by the framework. CP8: Ensure the framework includes tests for the optional behaviors defined in the specification. Explain how optional behaviors are accommodated by the framework. CP9: Ensure the test framework accommodates profiles, modules, product classes, and levels if they are used in the specification. For each dimension of variability used in the specification, demonstrate how the framework allows tests to be filtered by the dimension. CP10: Ensure the framework accommodates the conformance policy defined in the specification. Demonstrate how the framework allows tests to be filtered by levels. - CP7-10 are now combined in GL5 CP3 and GL5 CP1 CP11: Ensure the framework supports test results verification. Demonstrate results verification by testing three products. - Part of this relates to GL5 CP1. result verification is a good part of what makes a test suite runable in a consistent and predictibile way CP12: Ensure the framework allows the tests to be documented. Explain how to document the tests within the framework. - Also merged into GL5 CP3. CP13: Ensure the framework has proper test case management. Demonstrate how at least one of the following test case management functions is accomplished within the framework: managing additions, managing removals, filtering by various criteria - GL3 CP1 now. CP14: Ensure the framework allows specification coverage to be measured. Demonstrate the above by mapping a list of tests to the list of test assertions, grouped by areas - Also merged into GL5 CP3. Old G5: Provide the results reporting framework New G5: Test Execution Framework. CP1: Ensure that the test framework supports reporting the results. - Still here, no real change. CP2: Ensure the ease of use for results reporting. Demonstrate that the results reporting has sorting and filtering capabilities. - Still here. CP3: Document how the results reporting allows results to be exported in a self-contained format suitable to publication on the web. - Still here. CP4: Demonstrate that the results reporting provides details on failures (logs) sufficient to investigate. -Still here. CP5: Document how the results reporting allows for history/storing analysis of comments - Combined with G5 CP2. Old G6: Plan for tests development New G6: Result Reporting. CP1: Define priorities for the test areas - This has become part of G2, CP2. CP2: Introduce a mechanism for the early feedback on the test suite architecture and test framework. Document and publish the mechanism to the intended audience. - This has become the new G4 CP1. CP3: Ensure regular specification coverage analysis. Provide the schedule for specification coverage analysis. - Not here any more. To some degree this is covered in OpsGL by tying release schedule to test suite development. Specifically GL3 G7: Plan for conformance testing CP1: Document the plan for engaging vendors of implementations in conformance testing activities. CP2: Encourage Vendors to publish test results for their products by reserving a special space where information pertaining to test results can be maintained. - No changes.
Received on Thursday, 20 March 2003 19:11:52 UTC