- From: Lofton Henderson <lofton@rockynet.com>
- Date: Sun, 08 Jun 2003 14:50:12 -0600
- To: Lynne Rosenthal <lynne.rosenthal@nist.gov>
- Cc: www-qa@w3.org
At 07:56 AM 6/6/03 -0400, Lynne Rosenthal wrote: >Should specification developers be required to include Test Assertions >(TA) in their specifications? >[...] >This topic will be discussed on monday's (june 9), QA WG telecon. Lynne's question is a fundamental one that we need to resolve, but there are several other specific issues related to test assertions and requirements. The TA issue group for Monday is: LC-13, 14, 29.4, 65, 67, 73.9, 75.9. (Of course, if we decide to do away with GL14 and CP14.1, 14.2, then some of the following [proposed] resolutions change.) First, two "issues" that aren't actually in the LC issues list... Definitions ----- Many terms have occurred in these issues and email, and only one is defined (in QA Glossary or SpecGL "4. Definitions"): test assertion. Other terms used without definition: conformance requirement conformance-affecting statement testable statement test requirement conformance criteria testable sentences testable technical requirement We can probably get by with a subset of these terms, and we ought to have agreed definitions for that subset. Definitions sub-topic ----- There is a mail thread [1] that generated quite a bit of dialog on this topic, and I'm uncertain if there was any agreed conclusion: is "conformance requirement" (ConfReq) synonymous with "testable conformance requirement"? Alternatives: ** yes, it should be part of the definition of "conformance requirement" that a ConfReq is testable; ** not by definition, but SpecGL needs to say that any ConfReq MUST be testable; ** not by definition, but SpecGL needs to say that any ConfReq SHOULD be testable (i.e., this is a goal); LC-13 ----- points out that SpecGL fails CP14.1 by not providing Test Assertions. We already resolved that we'd like SpecGL to be AAA conformant eventually (CR). LC-14 ----- Asks if TAs can be in a separate document. Resolution (from Tokyo or Seattle f2f): Yes. LC-29.4 ----- "It might be valuable to explain some desirable characteristics of a specified technical requirement", and then three examples of such are suggested. I'm unclear exactly what IJ means by "specific technical requirement", and the scope. I assume that he means conformance requirement, and the scope is: SpecGL should include such stuff in its definition of the concept of conformance requirement, as such concept is applied (in SpecGLs CPs) to target specifications. Any case, after we sort out the meaning an scope, there are the three suggested attributes to consider: mutual independence; minimal; structure and content of presentation (of a "technical requirement"). LC-65 ----- (Resolved.) LC-67 ----- Two parts. The 2nd part -- suggested caveats and constraints on usage of RFC2119 "MUST" -- is resolved. The 1st part is still open: "Must all testable statements and or test assertions (sorry I'm not clear on the difference between them and maybe that needs to be clarified, too) contain RFC 2119 key words?". Hmmm... I think the answer is "no". If it were "yes", then UAAG10 for example would fail.... UAAG10 subscribes to the use of RFC2119, "This document demands substantially more conformance flexibility than can be achieved using the terms "must", "should", and "may" alone, as defined in RFC 2119 [RFC2119]. Where "must", "should", "required", and "may" appear in this document, they are used consistently with RFC 2119 for a chosen conformance profile." But it also says "The imperative voice (e.g., "Allow configuration ...") used in the checkpoint provisions implies "must"..." I don't think we want UAAG10 to fail CP13.1, do we? Proposal. We should clarify CP13.1 to indicate that RFC2119 be the principal way of expressing conformance requirements, but clearly defined supplementary (alternative?) terminology may be also used. LC-73.9 ----- Suggests some improvement of the definition of test assertion that occurs in the initial verbiage of GL14 [1]. Some of his re-wording is editorial, but he raises interesting questions about the TA definition: "Is this really mandatory for an assertion to be 'independent, complete, testable'. An assertion can't be completely independent of others. There can be assertion which are difficult to tests i.e. which can be treated as untestable." I would also question whether this suggested rewording is correct, "Each test assertion is an statement for requirements in the specification which can be tested independently." (It might need a little modification, to align better with our agreed distinction between "test assertion" and "conformance requirement"). LC-75.9 ----- Is SpecGL's checklist (and ICS) a list of test assertions? Suggested resolution. No. It's close, but each line in the checklist is the statement of the checkpoint, which is its title, and is not normative. The normative bits are found in the styled sections following the checkpoint statement, and are prefaced with: "Conformance requirements:". Note also that may be more than one ConfReq in this section, for a given checkpoint. Note. The block as a whole is marked up with 'class="TA"' [yes, yes ... it should be 'CR'.] So we could easily extract a list of Conformance Requirements with XSLT, or at least a list of blocks of CRs. (With a little more markup, we could tag each individual requirement in the block). Then someone (Mark!) could translate them all to TAs. (Unless we change SpecGL's GL14 requirements to "list of TAs or CRs".) ### end ### Regards, -Lofton. [0] http://www.w3.org/QA/WG/lc-issues [1] http://lists.w3.org/Archives/Public/www-qa/2003May/0007.html [2] http://www.w3.org/TR/qaframe-spec/#Gd-include-assertions
Received on Sunday, 8 June 2003 16:49:36 UTC