Annotated minutes from TestGL discussion in Crete

GL 1 deals with analysis of the test suite.
CP1.1 Define test suite scope
Remove sentence: "Note, a WG may have multiple test suites for different
parts"

 >> DONE

CP1.1 and CP1.3 Are these redundant?
No. CP1.1 refers to the overall scope/goal. CP1.3 refers to the test
methodology applied to the whole test suite as well as applied to
components (parts) of the test suite.

GL 2 Identify and tag testable assertions
CP2.1 Is it a requirement that TAs be developed? Yes. Need to reword
conformance requirements to indicate that the TAs must come from the spec
if they are there, but they may be somewhere else. Make P1

 >> DONE

CP2.2 Metadata must be associated with test assertions
Rationale is not really a rationale, but could be a title. Also, do we
want SHOULD or MUST.
Make P1. Current 2.2 title is the rationale. Make the title, "Metadata
MUST be associated with TAs." Need to make active voice. Require a
minimum set of required metadata. The rest of the metadata could be either
a SHOULD in this CP, a P2 checkpoint or in ExTech. Decided to have a P2
checkpoint with a second set of metadata. Will need to develop a schema
for this. Also, indicate that these sets of metadata in not exhaustive.

 >> DONE

GL3 test cases management systems
Note that we assume that the tests exist, we don’t discuss how they appear.
We go from TAs to managing the test cases. Should there be something
regarding the creation of tests. OpsGL addresses some of this. Missing a
guideline on performing QA on test cases do QA on your test materials.

 >> This is all covered in OpsGL (even QA - OpsGL includes the statement
 >> "the WG MUST define in its QA Process Document a procedure for reviewing
 >> test materials contributions, and the procedure MUST minimally address
 >> criteria for accuracy, scope, and clarity of the tests."
 >> How to handle this overlap?

CP3.2 rephrase to be similar to CP2.2, i.e., the notion of having a minium
set of metadata
CP3.4 Does this belong? This should reflect the storing of
results. Actually, it may be possible to remove, since this is obvious -
all test cases have an expected results. Saying this is O.K. Should this
be a separate checkpoint. Yes. Reword, You must have expected results
associated with each of your test cases. Make P1

 >> DONE

Do we need a better definition of a test case management system? The
concepts (functions) here are fuzzy, there is no clear line between what is
part of the management system and what is part of the framework. The
functions can be done in either systems. For example, selecting the
appropriate test cases can be done at build time (of the test suite) or at
run time (when executing the test suite). A concepts section would be
beneficial.

 >> TBD - transferred to issues list

GL4 Provide a test framework. (harness)
CP4.1 This should not be P1. The metadata and documentation enables the
development of a harness. A test harness is an (automated?) mechanism
that provides a consistent interface to testing. Harness refers to the
process of executing the tests.

 >> DONE

Missing checkpoints or a guideline on documentation. Documentation could
be part of the harness, but could also be part of other things, e.g.,
test cases

 >> TBD - transferred to issues list

CP4.2 Prototype the test framework
Prototype is the wrong word. A framework should not be built in
isolation. If you build a harness, then it should be beta tested and tried
out. Maybe relate this to the classes of products to ensure that the
harness runs on a wide variety of platfoms. The end of the rationale (run
on wide number of platforms) should be part of the conformance requirements.

 >> DONE

CP4.3 Automation of testing encouranged
Is this different from a harness, can paper be a test harness? A harness
provides a process (instructions) for executing the tests in a particular
sequence. Change.

Summary of GL4 discussion: Broadened into test execution process into 2
topics.
1. Document the process to execute tests,
2. If possible, it is desirable to automate. If automate then 4.2 follows.

 >> DONE

GL5 Test results
Merge CP5.1 and CP5.3. Record not just pass/fail, but other states that
apply.
There must be a well defined mechansim for reporting, this mechansim does
not have to be automated, but automation is desirable. Make the checkpoints
analagous to Guideline 4.

 >> DONE

GL6 Promote for conformance testing
Do we need this guideline since some of this is already in OpsGL. In
section 1.4 Relationship to other Framework documents, make it clear where
this picks up from OpsGL. Add CP6.1 to OpsGL, CP6.2 already in OpsGL.

Is it logical to divide into these 3 processes: test management system,
test framework and test reporting?

 >> Probably. See comments elsewhere about a Concepts section.

Within Test Materials, there are 3
partitions:Test Cases, Test Software, Test Documentation. Need to draft a
description of these partitions and provide a discussion of the licenses
with suggestions on how to apply the licenses, including a mention that
some things straddle the partitions.

 >> Licensing is covered in OpsGL. Another example of the overlap problem.

Review of Comments:
1. DM’s comments apply to a previous TestGL draft:
http://lists.w3.org/Archives/Public/www-qa-wg/2003May/0026.html
DM-1.1 Agree

 >> DONE (removed reference to multiple test suites). Should we
 >> address modularization?

DM1.2 Need to make sure we cover versioning.

 >> Checkpoint 3.2 of OpsGL coveres versioning. More overlap. Nevertheless,
 >> added version info to list of metadata

Should metadata tie back to SpecGL DoV? We talk about filtering, but
should filtering on the DoV allowed by the specification be called out?
Yes. Want to include or exclude tests based on DoVs.

 >> This is addressed in the metadata section.

Need to include a test purpose or description.

 >> DONE

GL4: talking about test case review. Can say something about where get
test data from put this is anaylsis guideline.

 >> Test case review is covered in OpsGL. More overlap

CP5.4: should there be mention of storage of results? Covered in test
results management.
We don’t mention anything about coverage. Need to add something about
coverage.

 >> DONE

2. SM’s comments 
http://lists.w3.org/Archives/Public/www-qa/2003May/0036.html
GL4: concepts have been clarified.
CP4.3 Agreed. Add a sentence to clarify.

 >> DONE

CP5.1 Need to clarify and define terms. Use some of the suggested words.

 >> DONE

3. PF’s comments
http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0031.html
1. Addressed
2. Agreed

 >> DONE

3. need to look at intro

 >> REVIEWED BELOW

4. Relates to coverage and strategy. Will say something about coverage.

 >> Coverage is now addressed, but the broader question of differing 
strategies
 >> at different points in the development cycle has not been addressed. 
Transferred
 >> this to issues list.

5, 6, 7. Agreed

 >> Ongoing review required.

Processing Plan.
WG draft by late September
Need ExTech document prior to first call
First Last Call: ??

====================

Review of Introduction
Disagreement on whether a scope can be a set of requrements. Class of
products also included interoperability. Intended audience, delete
conformance. Remove or move last paragraph of 1.4. Sections 1.5, 1.6, 1.7
make sure that wording is consistent with other documents. Need to create
use cases for TestGL.

Review of Conformance and Definitions
Definitions need work. Conformance Section needs to be reviewed.

 >> Not yet addressed. Transferred to issues list.

Received on Sunday, 17 August 2003 20:43:48 UTC