Re: TestGL: issues to discuss on Monday's telconf- Dec 9th

Some quick comments about TestGL discussion draft [1] ...

At 02:37 AM 12/6/02 -0800, Kirill Gavrylyuk wrote:

>The goal is to define what is critical to fix (other then some English 
>wording and broken linksJ) before TestGL goes to FPWD.

This is a key point.  Final text must be to the Webmaster in about 8 days 
(latest:  Wed, 12/18).  That leaves very little time for any substantive 
changes.  We may be constrained to focus on clarity, fix wording and links, 
and pubrules compliance.  In general, I think this will be fine for FPWD, 
given a little attention to those aspects.  It is at least as good as our 
FPWDs of OpsGL and SpecGL (probably better).

Global Comment -- Scope
=====

I think that the scope, target audience, focus, (and use scenarios) are not 
sufficiently clear.  Some of TestGL seems to deal with content, 
characteristics, and quality of the test materials.  Some of it still 
sounds like instructions for how to build the test materials (esp, GL1 and 
GL6 still sound this way).  I recall a past discussion about this -- that 
we would clarify (initially) whether we were dealing with TM content, 
how-to-build, or both.

I do not suggest to try to restructure or reword the GL or CP at this 
point.  Rather, perhaps we can just clarify what we are doing in this 
version.  I.e., work on introductory sections.

Current places that address the scope and purpose include:  Sec.1.0, 3rd 
pgph through 5th pgph; the roadmap of Sec 1.2, 2nd pgph through 4th 
pgph.  (Note.  Sec1.0, 3rd pgph, 1st sentence sounds like we are targeting 
the testing process, as opposed to content of test materials and process of 
building test materials.)

Kirill's specific questions:
=====
>    * Are the Priorities definition in the Introduction acceptable for FPWD?
1st:  I would suggest to use the same initial "importance keywords" as 
OpsGL and SpecGL, e.g., for P1 "Critical/essential".

2nd:  In P1, for example, see the statement "test materials do not 
verify".  This statement violates our own Conformance Disclaimer of section 
3.2.  It should not be so absolute.  I.e., it should say that there is 
"high probability that the test materials will not adequately verify...".  Etc.
>    * Are the definitions in the local Glossary acceptable?
I suggest to call it "Definitions", instead of Glossary (similar to SpecGL).

I would like to see a definition of "test framework" somewhere, and have 
that definition in this section also.  Since the concept is so integral to 
TestGL, the definition should be as verbose as necessary in order to 
clarify the concept and its boundaries, i.e., what is included in test 
framework and what is not.
>    * Are Ck1.5 1.9 in sync with the specification Guidelines?
>    * Is the Gd2 clear? Do checkpoints 2.1 and 2.2 verify the quality of 
> the test suite structure?
>    * Is the intent of the Gd3 clear? Is definition of the testing 
> methodology term is clear? Was the rewording of the checkpoints appropriate?
>    * Gd4: Does the Gd4 verify the test framework quality? Are Ck 4.8 4.10 
> in sync with the specification guidelines?
>    * Gd5. Is it reasonable to treat Results reporting as part of the Test 
> Framework?
IMO, yes.  One could argue, in the abstract, about whether it belongs in TF 
or not.  However, from practical experience, if the builders of the TM do 
not provide or specify at least one reporting mechanism, then the task 
might remain undone.
>    * Are the Gd6 & Gd7 acceptable for the FPWD? is the intent clear?
>    * Review Conformance clause
>    * General issue: do the checkpoints scare potential test suite 
> implementers?
>    * General issue: did the rewording help to target the checkpoints on 
> the quality of the test materials/test development/testing plan?
I think more clarification could be done here, as I suggested in "Global 
Comment".  Mostly, I'm suggesting (for now) that we be explicit somewhere 
that we are indeed targeting these three distinct areas (TM content and 
characteristics; how to write TM; how to apply TM.)

Global Editorial comments
======

There are lots of occurrences of the SLASH construct, "..this/that..".  The 
language should be expanded in these places.  (What does each occurrence 
mean:  alternative statement; synonymous statement; "and"; "or"; or 
"and/or"?).  Example, CP 5.5:  "Document how the results reporting allows 
for history/storing analysis comments."

Look for "compliance" and "comply" and change them to "conformance" and 
"conform".

Eliminate personal pronouns "I, we, you".  These are banned in the W3C 
style manual, although they are still used a lot in W3C specs.  Minimally, 
"I" must go (see 3rd pgph of GL3).

Multi-sentence checkpoint statements.  In general, these should be avoided 
and the subsequent sentences ought to go into fulfillment criteria, etc.  I 
would queue this fix for *after* FPWD.

Regards,
-Lofton.

>[1] 
><http://www.w3.org/QA/WG/2002/12/qaframe-test-20021205.html>http://www.w3.org/QA/WG/2002/12/qaframe-test-20021205.html 
>