other TA issues [was: Re: Should Test Assertions be required]

At 07:56 AM 6/6/03 -0400, Lynne Rosenthal wrote:
>Should specification developers be required to include Test Assertions 
>(TA) in their specifications?
>[...]
>This topic will be discussed on monday's (june 9), QA WG telecon.

Lynne's question is a fundamental one that we need to resolve, but there 
are several other specific issues related to test assertions and 
requirements.  The TA issue group for Monday is:  LC-13, 14, 29.4, 65, 67, 
73.9, 75.9.

(Of course, if we decide to do away with GL14 and CP14.1, 14.2, then some 
of the following [proposed] resolutions change.)

First, two "issues" that aren't actually in the LC issues list...

Definitions
-----
Many terms have occurred in these issues and email, and only one is defined 
(in QA Glossary or SpecGL "4. Definitions"):  test assertion.  Other terms 
used without definition:

conformance requirement
conformance-affecting statement
testable statement
test requirement
conformance criteria
testable sentences
testable
technical requirement

We can probably get by with a subset of these terms, and we ought to have 
agreed definitions for that subset.

Definitions sub-topic
-----
There is a mail thread [1] that generated quite a bit of dialog on this 
topic, and I'm uncertain if there was any agreed conclusion:  is 
"conformance requirement" (ConfReq) synonymous with "testable conformance 
requirement"?  Alternatives:

** yes, it should be part of the definition of "conformance requirement" 
that a ConfReq is testable;
** not by definition, but SpecGL needs to say that any ConfReq MUST be 
testable;
** not by definition, but SpecGL needs to say that any ConfReq SHOULD be 
testable (i.e., this is a goal);

LC-13
-----
points out that SpecGL fails CP14.1 by not providing Test Assertions.  We 
already resolved that we'd like SpecGL to be AAA conformant eventually (CR).

LC-14
-----
Asks if TAs can be in a separate document.  Resolution (from Tokyo or 
Seattle f2f):  Yes.

LC-29.4
-----
"It might be valuable to explain some desirable characteristics of a 
specified technical requirement", and then three examples of such are 
suggested.  I'm unclear exactly what IJ means by "specific technical 
requirement", and the scope.  I assume that he means conformance 
requirement, and the scope is:  SpecGL should include such stuff in its 
definition of the concept of conformance requirement, as such concept is 
applied (in SpecGLs CPs) to target specifications.

Any case, after we sort out the meaning an scope, there are the three 
suggested attributes to consider:  mutual independence; minimal; structure 
and content of presentation (of a "technical requirement").

LC-65
-----
(Resolved.)

LC-67
-----
Two parts.  The 2nd part -- suggested caveats and constraints on usage of 
RFC2119 "MUST" -- is resolved.

The 1st part is still open:  "Must all testable statements and or test 
assertions (sorry I'm not clear on the difference between them and maybe 
that needs to be clarified, too) contain RFC 2119 key words?".  Hmmm...  I 
think the answer is "no".  If it were "yes", then UAAG10 for example would 
fail....

UAAG10 subscribes to the use of RFC2119, "This document demands 
substantially more conformance flexibility than can be achieved using the 
terms "must", "should", and "may" alone, as defined in RFC 2119 [RFC2119]. 
Where "must", "should", "required", and "may" appear in this document, they 
are used consistently with RFC 2119 for a chosen conformance 
profile."   But it also says "The imperative voice (e.g., "Allow 
configuration ...") used in the checkpoint provisions implies "must"..."

I don't think we want UAAG10 to fail CP13.1, do we?

Proposal.  We should clarify CP13.1 to indicate that RFC2119 be the 
principal way of expressing conformance requirements, but clearly defined 
supplementary (alternative?) terminology may be also used.

LC-73.9
-----
Suggests some improvement of the definition of test assertion that occurs 
in the initial verbiage of GL14 [1].  Some of his re-wording is editorial, 
but he raises interesting questions about the TA definition:  "Is this 
really mandatory for an assertion to be 'independent, complete, testable'. 
An assertion can't be completely independent of others. There can be 
assertion which are difficult to tests i.e. which can be treated as 
untestable."

I would also question whether this suggested rewording is correct, "Each 
test assertion is an statement for requirements in the specification which 
can be tested independently."  (It might need a little modification, to 
align better with our agreed distinction between "test assertion" and 
"conformance requirement").

LC-75.9
-----
Is SpecGL's checklist (and ICS) a list of test assertions?

Suggested resolution.  No.  It's close, but each line in the checklist is 
the statement of the checkpoint, which is its title, and is not 
normative.  The normative bits are found in the styled sections following 
the checkpoint statement, and are prefaced with:  "Conformance 
requirements:".  Note also that may be more than one ConfReq in this 
section, for a given checkpoint.

Note.  The block as a whole is marked up with 'class="TA"' [yes, yes ... it 
should be 'CR'.]  So we could easily extract a list of Conformance 
Requirements with XSLT, or at least a list of blocks of CRs.  (With a 
little more markup, we could tag each individual requirement in the 
block).  Then someone (Mark!) could translate them all to TAs.  (Unless we 
change SpecGL's GL14 requirements to "list of TAs or CRs".)
### end ###

Regards,
-Lofton.

[0] http://www.w3.org/QA/WG/lc-issues
[1] http://lists.w3.org/Archives/Public/www-qa/2003May/0007.html
[2] http://www.w3.org/TR/qaframe-spec/#Gd-include-assertions

Received on Sunday, 8 June 2003 16:49:36 UTC