W3C home > Mailing lists > Public > www-qa-wg@w3.org > March 2003

Action Item AI-20030310-1

From: pfawcett <pfawcett@speakeasy.org>
Date: Thu, 20 Mar 2003 16:11:57 -0800
To: www-qa-wg@w3.org
Message-Id: <BABFB3EE-5B31-11D7-844B-000393A772C4@speakeasy.org>

Ok, here is a stab at trying to map the old Test GL contents to the new 
Test GL outline.
I've sort of done it in two parts, the first part is a simple mapping 
with out any comments but it wont make much sense unless your looking 
at the actual TestGL document and the new outline. The second part has 
the actual checkpoint titles which makes it easier but it gets hard to 
keep perspective of the 'big picture'. The two together do an ok job of 
giving the over all idea. Most of what was there before is still there, 
there are only a few checkpoints that I could not find any mapping for. 
Many checkpoints are merged into one in the new document (specifically 
GL1 CP2-9 and GL4 CP4,CP6,CP7-10,CP12 and CP14).

Patrick if you think I missed anything your welcome to add stuff.

If nothing else this is a good basis for discussion on monday.

Peter


The summary is:
Old Outline     New Outline
--------------------------------------
GL1 CP1     -> Stays (GL1 CP1).
GL1 CP2-9  -> GL2 CP1
GL1 CP10   -> Stays (GL1 CP4).

GL2 CP1-2  -> GL1 CP2 and GL3 CP2

GL3 CP1    -> GL1 CP3
GL3 CP2    -> Moved to extec and/or descriptive language.

GL4 CP1    -> Moved to extec and/or descriptive language.
GL4 CP2    -> Moved to extec and/or descriptive language.
GL4 CP3    -> Combined with GL6 CP2.
GL4 CP4    -> Part of GL5 CP3
GL4 CP5    -> Part of GL5 CP1
GL4 CP6    -> Related to GL5 CP3 and also OpsGL
GL4 CP7-10 -> Combined into G5 CP3
GL4 CP11   -> Not sure if this makes is anywhere....
GL4 CP12   -> Combined into G5 CP3
GL4 CP13   -> Merged into GL3 CP1
GL4 CP14   -> Combined into G5 CP3

GL5 CP1    -> Stays.
GL5 CP2    -> Stays.
GL5 CP3    -> Stays.
GL5 CP4    -> Stays.
GL5 CP5    -> Stays but is combined with CP2.

GL6 CP1    -> GL2 CP2
GL6 CP2    -> GL4 CP1
GL6 CP3    -> Removed, may already be covered by OpsGL.

GL7        -> Stays the same.


More in depth analysis:

Old G1:	Provide analysis of the specification(s).
New G1: High level analysis of specification.

	CP1: Identify the target set of specifications that are being tested.
	- CP1 is slightly reformed but is still here. It is now included as 
part of
		the scope of the test suite (likely still checkpoint 1 too).

	CP2: Identify test assertions from the target set of specifications.
	CP3: Group test assertions by levels, profiles, modules, etc.
	CP4: Identify those test assertions that are part of the
		conformance criteria
	CP5: Identify those test assertions that are part of the conformance
		criteria
	CP6: Identify optional behaviors in the specification
	CP7: Identify behaviors that are undefined or defined ambiguously
		in the specification
	CP8: Identify explicitly undefined behaviors in the specification
	CP9: Identify contradictory behaviors in the specification
	- CP2-9 combine to become the first checkpoint of the new GL2.
	
	CP10: List the user scenarios for the specification
	- CP10 stays here.

Old G2: Declare the structure of the test suite.
New G2: Low level analysis of specification.

	CP1: Document the structure for the test suite.
	CP2: Provide mapping between the test suite structure and the
		specification structure.
	- CP1-2 are combined into the CP2 of GL1. Part of CP2 is also now
		in the new GL3, CP2 of the outline, in that a mapping of test case
		to assert to normative language is one of the pieces of metadata
		that system should support.
		

Old G3: Document the testing methodology.
New G3: Provide A Test Management System

	CP1: For each test area, identify the testing approach
	- CP1 This is now GL1 CP2 or 3. What is being tested, how much, how,
		what priority.

	CP2: Identify publicly available testing techniques. List the
		publicly available testing techniques that have been reused
	- CP2 is moved to extec or descriptive language

Old G4: Provide the test automation and framework
New G4: Test Suite Development

	CP1: List available test frameworks and applicable automation.
		Identify available test frameworks used. If none, justify
		why new frameworks are needed and existing ones could not be used.
	- CP1 is moved to extec or descriptive language

	CP2: Ensure the framework and automation are platform independent.
		Demonstrate on 3 platforms. Ensure that the framework and automation
		are built using open standards.
	- CP2 is moved to extec or descriptive language

	CP3: Ensure the framework and automation are applicable to any
		product or content that implements the specification. Demonstrate
		with three products or contents. Ensure that the framework and
		automation are built using open standards.
	- CP3 is still here but focus is on prototyping and ensuring that the 
framework
		is suitable for it's intended purpose.
		
	CP4: Ensure the framework makes it easy to add tests for any of the
		specification areas. Demonstrate, through an example, how tests
		are added to a specification area.
	- specification area is one of the meta data criteria that the test 
management
	system should support, so this get's combined into GL5 CP3
	
	CP5: Ensure the ease of use for the test automation. Document how
		the test automation is used.
	- Part of GL5 CP1
	
	CP6: Ensure the framework allows for specification versioning and
		errata levels. Explain how specification versioning and errata
		levels are accommodated by the test framework
	- Not here except as one of the meta data criteria. Explicit 
versioning and
	eratta guidelines also exist in OpsGL.

	CP7: Ensure the framework accounts for choices allowed by discretionary
		choices in the specification. Explain how discretionary behaviors
		are accommodated by the framework.
	CP8: Ensure the framework includes tests for the optional behaviors
		defined in the specification. Explain how optional behaviors are
		accommodated by the framework.
	CP9: Ensure the test framework accommodates profiles, modules,
		product classes, and levels if they are used in the
		specification. For each dimension of variability used in the
		specification, demonstrate how the framework allows tests to be
		filtered by the dimension.
	CP10: Ensure the framework accommodates the conformance policy defined
		in the specification. Demonstrate how the framework allows tests
		to be filtered by levels.
	- CP7-10 are now combined in GL5 CP3 and GL5 CP1

	CP11: Ensure the framework supports test results verification.
		Demonstrate results verification by testing three products.
	- Part of this relates to GL5 CP1. result verification is a good part 
of what makes
	a test suite runable in a consistent and predictibile way
		
	CP12: Ensure the framework allows the tests to be documented. Explain
		how to document the tests within the framework.
	- Also merged into GL5 CP3.
	
	CP13: Ensure the framework has proper test case management.
		Demonstrate how at least one of the following test case management
		functions is accomplished within the framework: managing additions,
		managing removals, filtering by various criteria
	- GL3 CP1 now.
	
	CP14: Ensure the framework allows specification coverage to be
		measured. Demonstrate the above by mapping a list of tests to the
		list of test assertions, grouped by areas
	- Also merged into GL5 CP3.

Old G5: Provide the results reporting framework
New G5: Test Execution Framework.

	CP1: Ensure that the test framework supports reporting the results.
		- Still here, no real change.

	CP2: Ensure the ease of use for results reporting. Demonstrate that
		the results reporting has sorting and filtering capabilities.
		- Still here.
		
	CP3: Document how the results reporting allows results to be exported
		in a self-contained format suitable to publication on the web.
		- Still here.

	CP4: Demonstrate that the results reporting provides details on
		failures (logs) sufficient to investigate.
		-Still here.

	CP5: Document how the results reporting allows for history/storing
		analysis of comments
		- Combined with G5 CP2.

Old G6: Plan for tests development
New G6: Result Reporting.

	CP1: Define priorities for the test areas
		- This has become part of G2, CP2.
		
	CP2: Introduce a mechanism for the early feedback on the test suite
		architecture and test framework. Document and publish the mechanism
		to the intended audience.
		- This has become the new G4 CP1.
		
	CP3: Ensure regular specification coverage analysis. Provide the
		schedule for specification coverage analysis.
		- Not here any more. To some degree this is covered in OpsGL
		by tying release schedule to test suite development. Specifically GL3

G7: Plan for conformance testing

	CP1: Document the plan for engaging vendors of implementations in
		conformance testing activities.
	CP2: Encourage Vendors to publish test results for their products by
		reserving a special space where information pertaining to test
		results can be maintained.
	- No changes.
Received on Thursday, 20 March 2003 19:11:52 GMT

This archive was generated by hypermail 2.2.0 + w3c-0.30 : Thursday, 9 June 2005 12:13:13 GMT