Re: Should Test Assertions be required

I think the real question is being missed.  It is:  "Who should develop the 
test assertions - the spec developers or the test suite developers?"  Once 
we decide who this should be, we'll have our answer.

There are pros and cons for each of these, as well.

Mark



At 12:46 PM 6/6/2003 -0400, david_marston@us.ibm.com wrote:

>Lynne Rosenthal instiagted this thread by writing:
> >Should specification developers be required to include Test Assertions
> >(TA) in their specifications?
> >some Pros:
> >1. ensures testable specifications
> >2. consensus in what is required
> >3. facilitates test generation
>
>To which I would add:
>4. useful for precise citations (by test cases, emails discussing fine
>    points, etc.)
>
> >some Cons:
> >1. cost (time) to create
> >2. may not be appropriate or adequate for test developer's use...
> >3. not needed since generate tests directly from Spec....
>[#3 is actually from a schema of the markup language, for those specs
>that define an XML vocabulary.]
>
> >IMO specifications should (must?) identify the testable statements or
> >test requirements.
>
>To me, the above is the very epitome of a Priority 2 requirement. It's
>a substantial benefit to have them, not just "nice to have" but also not
>vital to the task of writing test cases. Right now, we manage to isolate
>testable passages in the current specs.
>
> >But depending on your definition of TA, these may not be as formally
> >represented (stated) as a TA.
>
>Strongly agree. Given the current spec-writing practices, even simple
>tags surrounding testable sentences ("virtual highlighter") would fill
>a lot of the needs. Some spec editors (and WG members who get cajoled
>into writing substantial amounts of verbiage) may not buy in to having
>separate passages full of TAs, but they may be willing to put tags
>around the sentences that they were writing anyway.
>
>I think that the W3C's spec-writing practices will improve over time.
>The 1.0 Spec Guidelines need to address the weaknesses that allow
>unintended varying interpretations (see Pro#2 above) and related
>threats to interoperability. Most of the Pros and Cons listed here
>are not exclusive to TAs; we can achieve better testability and
>consensus without TAS; but each technique that contributes to the goals
>can be measured in a cost/benefit way relative to the other techniques.
>The 2.0 Spec Guidelines can push for a higher level, possibly even
>building on new designs for machine-processable assertions. (I have no
>guess about a year when 2.0 might be necessary.)
>
>Overall, I would like to encourage spec editors to mark up passages
>that have the level of testability that TAs have. If they don't want to
>have visible distinctiveness for such passages, that's fine with me for
>the time being. I hope that some of the specs will be more adventurous.
>.................David Marston

****************************************************************
Mark Skall
Chief, Software Diagnostics and Conformance Testing Division
Information Technology Laboratory
National Institute of Standards and Technology (NIST)
100 Bureau Drive, Stop 8970
Gaithersburg, MD 20899-8970

Voice: 301-975-3262
Fax:   301-590-9174
Email: skall@nist.gov
****************************************************************

Received on Monday, 9 June 2003 10:38:04 UTC