Re: TestGL WD for review

Peter -- Here are some comments on the TestGL WD.

Section 1.5

In the following paragraph:

Each checkpoint is intended to be specific enough so that someone can 
implement the checkpoint as well as verify that the checkpoint has been 
satisfied. A checkpoint will contain at least one, and may contain multiple 
individual requirements, that use RFC2119 normative keywords. See the 
Conformance section for further discussion of requirements and test 
assertions.

Comment :Are you referring to section 3?. There is nothing related to "test 
assertion" in this section.

Guideline 5. Test execution

Comments:

1. Propose changing guideline title to "Test suite/cases Execution:" if the 
intention is to address the execution of individual tescases, a set of 
testcases or the whole testsuite. Also, to maintain consistency with  GL.4.

2. Checkpoint 5.1 : Test management system requirements should be addressed 
in GL 3.  The rationale could then be redefined by  explaining that the 
test cases must be precise in its intended purpose as prescribed in the 
test management system.

3. Checkpoint 5.2 : Automation of testing ins encouraged.
Is this checkpoint referring to automation of test suite/case execution? or 
automation of test generation? The conformance requirement and the 
rationale are contradicting each other. The conformance requirement is 
referring to a test harness to allow automation and the rationale is 
referring to test coverage and framework automation for automatic test 
generation.
This GL is suppose to be dealing with the execution of tests, test coverage 
belong in a different GL.  Also, do we have a definition for test harness? 
this definition could be include in the definition section of the document.

4. Checkpoint 5.3: Test management system requirements should be addressed 
in GL 3.
5. Checkpoint 5.4: Again this checkpoint is also providing test management 
system requirements.

Guideline 6
1. checkpoint 6.1
  Rationale: In the case of inter-operability test suites required for 
verifying two implementations, providing a system for result reporting 
allows implementors to submit uniform sets of results which helps ease the 
process of comparison and the determination of what features are in danger 
of being cut due to their lack of support.

Comment : The first sentence in the rationale is not clear.

2. Checkpoint 6.2

The rationale and the conformance requirement are getting me confused. The 
conformance requirement addressed the result reporting framework as 
allowing the generation of a report that could be published in the web and 
the rationale is referring to the type of report "unified web page or 
packages of pages".

3. Checkpoint 6.4

Just a recommendation for the rationale:

Rationale: Test results should be filterable based on a particular 
criteria. Filtering allow implementors to focus only on the results that 
satisfy a target criteria  while being able to exclude tests that are not 
applicable to a given implementation.


General comments: Are you planning to include aspects related to test suite 
publication - cover page development for public use?

Cordially,

Sandra




At 09:32 AM 4/25/2003 -0700, pfawcett wrote:
>Howdy folks,
>Sorry this is late.
>Here is the latest update of TestGL (both transformed and not) for
>internal review. Some sections are still missing I'm sorry to say but
>please focus on the entirely new introduction (I think it still needs
>work...), GL 5-7 (GL5 and 6 are almost totally new, GL7 is a
>reformatting and restructuring.) The glossary was moved to it's new
>location.
>And I tried to bring as much of the document into line as I could with
>the other documents in the family.
>Please send any comments you have as I'm working toward the next draft
>cause I do intend to make the next deadline of next week. The sooner I
>get peoples comments, the sooner I can get them in to the document.
>
>Thanks,
>Peter
>
>--
>pfawcett <pfawcett@speakeasy.org>
>Content-Disposition: attachment; filename=qaframe-test-20030423.html
>Content-Type: text/html; name=qaframe-test-20030423.html; charset=ISO-8859-1
>X-MIME-Autoconverted: from 8bit to quoted-printable by hawksbill.nist.gov 
>id h3PGfT8n016294
>
>

Sandra I. Martinez
National Institute of Standards and Technology
100 Bureau Drive, Stop 8970,
Gaithersburg, Md. 20899

(301) 975-3579
sandra.martinez@nist.gov

Received on Monday, 28 April 2003 16:05:34 UTC