Re: Should Test Assertions be required

Mark Skall wrote:
>>I think the real question is being missed. It is: "Who should develop 
>>the test assertions - the spec developers or the test suite 
>>developers?" Once we decide who this should be, we'll have our answer.

Lofton Henderson replied:
>So ... if we were to answer that TS developers should provide them, does 
>that mean that GL14 and its CPs go away?
[i.e., that SpecGL doesn't need to require TAs, because it only defines
what has to be in the spec, and the TAs would exist outside the spec]
>the Or does it just mean that someone else does the labor of generating
>the TAs?

I think any normative written product of the WG should be subject to
whatever part of SpecGL is applicable. We want the TAs to be a public
and normative item, endorsed by the WG if not written by them. Thus, GL
14 would still be applicable. I also think that someday, not
necessarily when SpecGL 1.0 is in effect, every spec that is testable
should have some form of in-depth evaluation by TS developers who work
only from the written spec. This would be a practical test of the
clarity and testability of the spec, just as pilot implementations test
the clarity and implementability of the spec.

Having a body outside the spec-writing body (outside the whole WG or just
outside the editorial task force) write TAs from the spec is one way to
test the spec's testability. But if insiders wrote the TAs, some other
task force could test the spec's testability in another way. Would we
want some of the test cases to be written by people other than those who
wrote (or generated) the TAs, as a check on the TAs?

I see the test materials (test suite) as the end, and TAs as a means.
BNF grammars and schemas are other formal notations, already present in
some W3C specs, are different means to make testable assertions. It is
more important to get the test cases from the formal expression of the
rules than to recast the rules into a uniform system of test assertions,
at least at this stage (2003) of our organizational maturity. Each test
case should cite the rules it purports to test, be they TAs, numbered
productions in BNF, or even just testable sentences.
.................David Marston

Received on Monday, 9 June 2003 13:02:16 UTC