Re: Should Test Assertions be required

Lynne Rosenthal instiagted this thread by writing:
>Should specification developers be required to include Test Assertions
>(TA) in their specifications?
>some Pros:
>1. ensures testable specifications
>2. consensus in what is required
>3. facilitates test generation

To which I would add:
4. useful for precise citations (by test cases, emails discussing fine
   points, etc.)

>some Cons:
>1. cost (time) to create
>2. may not be appropriate or adequate for test developer's use...
>3. not needed since generate tests directly from Spec....
[#3 is actually from a schema of the markup language, for those specs
that define an XML vocabulary.]

>IMO specifications should (must?) identify the testable statements or
>test requirements.

To me, the above is the very epitome of a Priority 2 requirement. It's
a substantial benefit to have them, not just "nice to have" but also not
vital to the task of writing test cases. Right now, we manage to isolate
testable passages in the current specs.

>But depending on your definition of TA, these may not be as formally
>represented (stated) as a TA.

Strongly agree. Given the current spec-writing practices, even simple
tags surrounding testable sentences ("virtual highlighter") would fill
a lot of the needs. Some spec editors (and WG members who get cajoled
into writing substantial amounts of verbiage) may not buy in to having
separate passages full of TAs, but they may be willing to put tags
around the sentences that they were writing anyway.

I think that the W3C's spec-writing practices will improve over time.
The 1.0 Spec Guidelines need to address the weaknesses that allow
unintended varying interpretations (see Pro#2 above) and related
threats to interoperability. Most of the Pros and Cons listed here
are not exclusive to TAs; we can achieve better testability and
consensus without TAS; but each technique that contributes to the goals
can be measured in a cost/benefit way relative to the other techniques.
The 2.0 Spec Guidelines can push for a higher level, possibly even
building on new designs for machine-processable assertions. (I have no
guess about a year when 2.0 might be necessary.)

Overall, I would like to encourage spec editors to mark up passages
that have the level of testability that TAs have. If they don't want to
have visible distinctiveness for such passages, that's fine with me for
the time being. I hope that some of the specs will be more adventurous.
.................David Marston

Received on Friday, 6 June 2003 12:52:34 UTC