What GL tests might look like

At 04:49 PM 12/18/03 -0500, Karl wrote:
>[...]
>         Karl: So I propose that at this end of the telconf we take one 
> test assertion, and try to design a test case for it, and see how it works.
>         Lofton: What do we do for Ops GL?

This isn't our primary topic today (telecon), but if we finish our TA-by-TA 
review ... here are some thoughts I been having for a while.

One reason that I asked my OpsGL question:  I am uncertain, for the case of 
OpsGL and for SpecGL, what a list of TAs really adds (like clarity, 
completeness, traceability, rigor, ...) to the end results -- the Test 
Materials for OpsGL.

That said, I do believe that our motivation in for requiring in **SpecGL** 
that TAs be generated (CP10.1 & 10.2) is sound -- that the process of 
producing them improves the specification.  We saw this process at work on 
Thursday -- we improved the quality of some of the ConfReqs during the 
exercise of trying to extract rigorous TAs from them.

(Btw, our SpecGL rationale for TAs is different from our rationale for 
TestGL requirements about TAs -- that they are an essential adjunct to Test 
Materials -- and this is why we have requirements about TAs in both of the 
GL documents).

As I look at OpsGL ConfReqs, I see that there are some problems that need 
to be addressed, and I think OpsGL will benefit from the same sort of TAs 
analysis.  Sweeping that aside for the moment...

The question I want to ask is about the end result, the OpsGL test suite 
(TS).  Is a TS based on derived TAs inherently much better than one that 
links directly to the ConfReqs?  I.e., if we could have the latter for 
OpsGL CR version in a week, while it would take us some months to do the 
work for the former ... should we wait for "the perfect", or go forward 
with "the good" as an interim step?

Let's look at an example, OpsGL CP4.4 ("Specify means for QA-related 
communication") [1].  It has two conformance requirements.  I'll look at 
the possibility of tests based directly on the ConfReqs, versus tests based 
on TAs derived from the ConfReqs.  The ConfReqs for CP4.4 are:

ConfReqs:

"
     *  the Working Group's QA Process Document MUST specify at least one 
public archived mailing list for QA announcements and submission of public 
QA comments;
     * the Working Group's QA Process Document MUST establish a publicly 
readable "Test" Web page.
"

=============== start ====================
I.) ConfReq-based test(s):

One could imagine these two tests in the test materials, which taken 
together test CP4.4:

     [@@Link(s) to OpsGL Spec@@]

     *  CR4.4-1:  the Working Group's QA Process Document MUST specify at 
least one public archived mailing list for QA announcements and submission 
of public QA comments;

     TS4.4-1, determine the appropriate choice:
     ____ The WG does not satisfy this requirement.
     ____ The WG satisfies this requirement, as demonstrated at (URL) 
_______________ .

     *  CR4.4-2:  the Working Group's QA Process Document MUST establish a 
publicly readable "Test" Web page.

     TS4.4-2, determine the appropriate choice:
     ____ The WG does not satisfy this requirement.
     ____ The WG satisfies this requirement, as demonstrated at (URL) 
_______________ .

============== end =====================

=============== start ====================
II.) TA-based test(s):
=====

First, derive this/these TAs from the ConfReqs, in the style that Mark did 
for SpecGL.  [Note.  MS tends to take multiple individual CRs and compound 
them into a single composite TA.  The other option, if the CRs for a CP are 
well-separated like in OpsGL, would be to have multiple atomic TAs].

     * TA4.4:  The WG's QAPD specifies at least one public mailing list for 
QA announcements and submission of public QA comments, AND, the WG's QAPD 
establishes a publicly readable "Test" Web page.

[Note again.  I think this could be factored into two independent, atomic 
TAs, TA4.4-1 and TA4.4-2.]

One could imagine the following in the OpsGL test materials, which would 
test CP4.4:

     [@@Link(s) to OpsGL Spec@@]

     TS4.4-1:  determine the appropriate choice:
      ____ The WG's QAPD specifies a public mailing list at (URL) 
________________
      ____ The WG's QAPD does NOT specify a public mailing list at.

     TS4.4-2:  determine the appropriate choice:

      ____ The WG's QAPD establishes a publicly readable "Test" Web page at 
(URL) _____________________.
      ____ The WG's QAPD does NOT establishes a publicly readable "Test" 
Web page.

[Note.  I factored, to make each line/TA/test simpler and clearer.]

=============== end ====================

Discussion
=====

Since our GL testing will be executed mostly by humans (low automation 
potential), it is not clear to me that the TA-based testing (II) is much 
superior to the ConfReq-based testing (I).  One interesting thing about (I) 
is that it could be extracted trivially and automatically from OpsGL by 
XSLT -- it is expedient.

The essential question:  is it better to offer (I) soon and replace it with 
(II) in some months?  or to wait and offer (II) in some months.

As I disclaimed, in making the "good now, perfect later" assessment I'm 
looking *only* at the end result.  I.e., it is based only on the usability 
(and traceability, etc) of the resulting tests.

It ignores the possible benefits of TA extraction and TA-based tests:

1.) clarification of testable ambiguities that might exist in the 
(human-friendly) ConfReqs.  (This benefit could be partially obtained by 
improvements to ConfReqs, as part of a careful reading for testability.)

2.) filling in missing bits, that are needed for testability, and that may 
(improperly?) be found in the context of the ConfReqs (e.g., in 
"Discussion", etc).  (This benefit could also be partially, though IMO not 
completely, obtained by a testability review of the ConfReqs.).

3.) I think we have started to conclude, in Thursday's SpecGL TA 
discussion, that we don't want to make the ConfReqs as rigorous as the TA, 
because we want the GL document to be readable and the ConfReqs are to be 
more human-friendly.  I'm not sure what impact this has on the ultimate 
potential quality and goodness of ConfReq-based test formulation.

All for now,
-Lofton.

[1] 
http://www.w3.org/TR/2003/CR-qaframe-ops-20030922/guidelines-chapter#Ck-proc-describe-comm

Received on Monday, 22 December 2003 09:33:03 UTC