Re: Should Test Assertions be required

At 07:56 AM 6/6/03 -0400, Lynne wrote:

>Should specification developers be required to include Test Assertions 
>(TA) in their specifications?

Small nit -- GL14 and the checkpoints CP14.1 and 14.2 say "provide".  As I 
recall we changed the wording from "include", at Tokyo or Seattle f2f, 
specifically to allow TAs to be in a companion document (which -- if I am 
remembering correctly -- also answers LC-14)


>[...snip pros & cons...]
>IMO specifications should (must?) identify the testable statements or test 
>requirements.

I agree.  Clear identification of the spec's conformance requirements is 
the overall gist and principal motive of all of the checkpoints in GL13 
([1]), "Clearly identify conformance requirements."  Identification of the 
individual requirements is a component of that overall goal.

>But depending on your definition of TA, these may not be as formally 
>represented (stated) as a TA.
>
>Lets think about the objective - identifying the testable statements and 
>facilitate the building of test suites - without going overboard and 
>burdening the specification developer.

After reading this, and the replies of Alex and David (who seem overall to 
agree on the lesser importance of TAs generation, relative to clear 
conformance requirements), I'm unclear what you (all) think should be done 
to SpecGL, GL14:

a.) remove it?
b.) keep it and lower it's priorities?
c.) revise it to account for different scenarios (below)?
d.) revise it to address/require "test assertions or conformance requirements"?

My thoughts...

I originally was a supporter of the requirement that spec writers should 
provide TAs.  Since Tokyo (Oct'02), we have refined our definition and 
understanding of TAs, and have generated some good examples from SpecGL 
itself -- for GL3 and its CPs, Mark generated a list of all 
the  conformance requirements (embedded in the text of SpecGL), and for 
each ConfReq, what an associated (derived) TA might look like.

Other messages in this thread show that there are really several scenarios 
for ultimately getting to test cases (TCs), starting from the specification 
text.  For example:

1.) formal-language text -> direct generation of TCs
2.) prose or formal text w/ TAs embedded -> extract TAs -> make TCs
3.) prose text w/ ConfReqs -> derive TAs -> make TCs

DOM and Schema are examples of #1 -- no separate TAs are involved.  SpecGL 
itself is an example of #3 -- clearly identified conformance requirements 
from which associated TAs could be derived (note that there are also 
examples of #3 where the ConfReq are not so clearly identified, and one 
must work the text to tease them out).  I can't think immediately of 
examples of #2, but I'm sure there are some -- in the text of the 
specification, the form of expression of conformance requirements qualifies 
them as test assertions.

Bottom line.  I don't think we want to force those in scenario #1 to 
generate a bunch of useless TAs.  On the other hand, I see the value of the 
TA generation in some cases, in scenarios #2 and #3.  I'm not thinking 
about the value to test builders, but rather the value to the specification 
goodness.  (And note -- we have already decided that we won't try to force 
the choice between scenario #2 and #3, for those who aren't using 
formalisms like #1.)

Recommendation about GL14.  Nothing specific yet.  Conceptually, I'd prefer 
to see TA considerations remain in some form, so that the benefits are 
recognized (for appropriate scenarios); but also the legitimacy of the 
different scenarios needs to be recognized (which implies some exemptions 
to whatever TA checkpoints we might end up with).

[[
Side topic.  Who should generate TAs?  Alex commented in this thread about 
the pitfalls of relying on the spec authors for the TAs [2], and after some 
specific points he concluded,  "By forcing spec authors to do testing we 
assign an important task to the wrong people."

I have some sympathy with this, or more precisely with the proposition that 
value is added by having other people than the spec writers participate in 
TA writing/derivation.  I'm talking here about value/feedback to spec 
writers, not value to test suite builders.  David's follow-up reply makes 
an interesting suggestion, that there should be a clear scope/completeness 
disclaimer with any TAs that the spec-writers provide.
]]

Regards,
-Lofton.

[0] http://www.w3.org/QA/WG/lc-issues#x14
[1] http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/#Gd-support-conventions
[2] http://lists.w3.org/Archives/Public/www-qa/2003Jun/0008.html

Received on Sunday, 8 June 2003 16:46:15 UTC