draft minutes: F2F June 18 morning

Comments?


June 18, morning
Scribe: Lynne

QA WG expresses its Many thanks to FORTH, Prof Markatos and Anna Doxastaki 
for hosting the meeting. Also thanks to Prof. Markatos and Anna for hosting 
a wonderful feast. Many thanks to Anna for handling the local arrangements 
and for making sure that we had everything we needed to conduct a 
successful meeting.

Schedule for processing SpecGL
   Disposition of Comments:  Aug 1st
   Next Publication to WG:  Sept 1

TestGL discussion
Review and overview of TestGL document structure and content. Guildelines 
deal with analysis, structure, and management of the test suite. The term 
Test management system (#GL3) in refers to management of the test 
cases.  Change name to Test materials management system. ‘Test framework’ 
(GL#4) refers to the test harness.  Rather than the Plan for conformance 
testing (GL#6), change to Promote conformance testing.

GL 1 deals with analysis of the test suite.
CP1.1 Define test suite scope
Remove sentence: “Note, a WG may have multiple test suites for different 
parts… “

CP1.1 and CP1.3 Are these redundant?
No. CP1.1 refers to the overall scope/goal.  CP1.3 refers to the test 
methodology  applied to the whole test suite as well as applied to 
components (parts) of the test suite.

GL 2 Identify and tag testable assertions
CP2.1 Is it a requirement that TAs be developed?  Yes.  Need to reword 
conformance requirements to indicate that the TAs must come from the spec 
if they are there, but they may be somewhere else.  Make P1
CP2.2 Metadata must be associated with test assertions
Rationale is not really a rationale, but could be a title.  Also, do we 
want SHOULD or MUST.
Make P1.  Current 2.2 title is the rationale.  Make the title, “Metadata 
MUST be associated with TAs.”  Need to make active voice.   Require a 
minimum set of required metadata. The rest of the metadata could be either 
a SHOULD in this CP, a P2 checkpoint or in ExTech.  Decided to have a P2 
checkpoint with a second set of metadata.  Will need to develop a schema 
for this. Also, indicate that these sets of metadata in not exhaustive.

GL3 test cases management systems
Note that we assume that the tests exist, we don’t discuss how they appear. 
We go from TAs to managing the test cases.  Should there be something 
regarding the creation of tests.  OpsGL addresses some of this. Missing a 
guideline on performing QA on test cases  do QA on your test materials.

CP3.2 rephrase to be similar to CP2.2, i.e., the notion of having a minium 
set of metadata
CP3.4 Does this belong?  This should reflect the storing of 
results.  Actually, it may be possible to remove, since this is obvious - 
all test cases have an expected results.  Saying this is O.K.  Should this 
be a separate checkpoint.  Yes.  Reword, You must have expected results 
associated with each of your test cases.  Make P1

Do we need a better definition of a test case management system? The 
concepts (functions) here are fuzzy, there is no clear line between what is 
part of the management system and what is part of the framework. The 
functions can be done in either systems. For example, selecting the 
appropriate test cases can be done at build time (of the test suite) or at 
run time (when executing the test suite).  A concepts section would be 
beneficial.

GL4 Provide a test framework. (harness)
CP4.1 This should not be P1.  The metadata and documentation enables the 
development of a harness.  A test harness is an (automated?)  mechanism 
that provides a consistent interface to testing. Harness refers to the 
process of executing the tests.  Missing checkpoints or a guideline on 
documentation. Documentation could be part of the harness, but could also 
be part of other things, e.g., test cases

CP4.2 Prototype the test framework
Prototype is the wrong word.  A framework should not be built in 
isolation.  If you build a harness, then it should be beta tested and tried 
out. Maybe relate this to the classes of products to ensure that the 
harness runs on a wide variety of platfoms. The end of the rationale (rfun 
on wide number of platforms) should be part of the conformance requirements.

CP4.3 Automation of testing encouranged
Is this different from a harness, can paper be a test harness?  A harness 
provides a process (instructions) for executing the tests in a particular 
sequence.  Change.

Summary of GL4 discussion: Broadened into test execution process into 2 
topics.
1. Document the process to execute tests,
2. If possible, it is desirable to automate.  If automate then 4.2 follows.

GL5 Test results
Merge CP5.1 and CP5.3.  Record not just pass/fail, but other states that 
apply.
There must be a well defined mechansim for reporting, this mechansim does 
not have to be automated, but automation is desirable. Make the checkpoints 
analagous to Guideline 4.

GL6 Promote for conformance testing
Do we need this guideline since some of this is already in OpsGL. In 
section 1.4 Relationship to other Framework documents,  make it clear where 
this picks up from OpsGL. Add CP6.1 to OpsGL, CP6.2 already in OpsGL.

Is it logical to divide into these 3 processes: test management system, 
test framework and test reporting?  Within Test Materials, there are 3 
partitions:Test Cases, Test Software, Test Documentation. Need to draft a 
description of these partitions and provide a discussion of the licenses 
with suggestions on how to apply the licenses, including a mention that 
some things straddle the partitions.

Review of Comments:
1.  DM’s comments apply to a previous TestGL draft:
http://lists.w3.org/Archives/Public/www-qa-wg/2003May/0026.html
DM-1.1  Agree
DM1.2 Need to make sure we cover versioning. Should metadata tie back to 
SpecGL DoV? We talk about filtering, but should filtering on the DoV 
allowed by the specification be called out? Yes.  Want to include or 
exclude tests based on DoVs.  Need to include a test purpose or description.
GL4: talking about test case review.  Can say something about where get 
test data from  put this is anaylsis guideline.

CP5.4: should there be mention of storage of results?  Covered in test 
results management.
We don’t mention anything about coverage. Need to add something about 
coverage.

2. SM’s comments  http://lists.w3.org/Archives/Public/www-qa/2003May/0036.html
GL4: concepts have been clarified.
CP4.3 Agreed.  Add a sentence to clarify.
CP5.1 Need to clarify and define terms.  Use some of the suggested words.

3. PF’s comments 
http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0031.html
1.  Addressed
2.  Agreed
3. need to look at intro
4.  Relates to coverage and strategy.  Will say something about coverage.
5, 6, 7. Agreed

Processing Plan.
    WG draft by late September
    Need ExTech document prior to first call
    First Last Call: ??

Review of Introduction
Disagreement on whether a scope can be a set of requrements.  Class of 
products also included interoperability.  Intended audience, delete 
conformance. Remove or move last paragraph of 1.4.  Sections 1.5, 1.6, 1.7 
make sure that wording is consistent with other documents.  Need to create 
use cases for TestGL.

Review of Conformance and Definitions
Definitions need work.  Conformance Section needs to be reviewed.