- From: Lofton Henderson <lofton@rockynet.com>
- Date: Sun, 08 Jun 2003 14:46:51 -0600
- To: Lynne Rosenthal <lynne.rosenthal@nist.gov>
- Cc: www-qa@w3.org
At 07:56 AM 6/6/03 -0400, Lynne wrote: >Should specification developers be required to include Test Assertions >(TA) in their specifications? Small nit -- GL14 and the checkpoints CP14.1 and 14.2 say "provide". As I recall we changed the wording from "include", at Tokyo or Seattle f2f, specifically to allow TAs to be in a companion document (which -- if I am remembering correctly -- also answers LC-14) >[...snip pros & cons...] >IMO specifications should (must?) identify the testable statements or test >requirements. I agree. Clear identification of the spec's conformance requirements is the overall gist and principal motive of all of the checkpoints in GL13 ([1]), "Clearly identify conformance requirements." Identification of the individual requirements is a component of that overall goal. >But depending on your definition of TA, these may not be as formally >represented (stated) as a TA. > >Lets think about the objective - identifying the testable statements and >facilitate the building of test suites - without going overboard and >burdening the specification developer. After reading this, and the replies of Alex and David (who seem overall to agree on the lesser importance of TAs generation, relative to clear conformance requirements), I'm unclear what you (all) think should be done to SpecGL, GL14: a.) remove it? b.) keep it and lower it's priorities? c.) revise it to account for different scenarios (below)? d.) revise it to address/require "test assertions or conformance requirements"? My thoughts... I originally was a supporter of the requirement that spec writers should provide TAs. Since Tokyo (Oct'02), we have refined our definition and understanding of TAs, and have generated some good examples from SpecGL itself -- for GL3 and its CPs, Mark generated a list of all the conformance requirements (embedded in the text of SpecGL), and for each ConfReq, what an associated (derived) TA might look like. Other messages in this thread show that there are really several scenarios for ultimately getting to test cases (TCs), starting from the specification text. For example: 1.) formal-language text -> direct generation of TCs 2.) prose or formal text w/ TAs embedded -> extract TAs -> make TCs 3.) prose text w/ ConfReqs -> derive TAs -> make TCs DOM and Schema are examples of #1 -- no separate TAs are involved. SpecGL itself is an example of #3 -- clearly identified conformance requirements from which associated TAs could be derived (note that there are also examples of #3 where the ConfReq are not so clearly identified, and one must work the text to tease them out). I can't think immediately of examples of #2, but I'm sure there are some -- in the text of the specification, the form of expression of conformance requirements qualifies them as test assertions. Bottom line. I don't think we want to force those in scenario #1 to generate a bunch of useless TAs. On the other hand, I see the value of the TA generation in some cases, in scenarios #2 and #3. I'm not thinking about the value to test builders, but rather the value to the specification goodness. (And note -- we have already decided that we won't try to force the choice between scenario #2 and #3, for those who aren't using formalisms like #1.) Recommendation about GL14. Nothing specific yet. Conceptually, I'd prefer to see TA considerations remain in some form, so that the benefits are recognized (for appropriate scenarios); but also the legitimacy of the different scenarios needs to be recognized (which implies some exemptions to whatever TA checkpoints we might end up with). [[ Side topic. Who should generate TAs? Alex commented in this thread about the pitfalls of relying on the spec authors for the TAs [2], and after some specific points he concluded, "By forcing spec authors to do testing we assign an important task to the wrong people." I have some sympathy with this, or more precisely with the proposition that value is added by having other people than the spec writers participate in TA writing/derivation. I'm talking here about value/feedback to spec writers, not value to test suite builders. David's follow-up reply makes an interesting suggestion, that there should be a clear scope/completeness disclaimer with any TAs that the spec-writers provide. ]] Regards, -Lofton. [0] http://www.w3.org/QA/WG/lc-issues#x14 [1] http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/#Gd-support-conventions [2] http://lists.w3.org/Archives/Public/www-qa/2003Jun/0008.html
Received on Sunday, 8 June 2003 16:46:15 UTC