Re: Spec Guideline: subjectivity (re: issue #91)

On Mon, 7 Oct 2002, David Marston/Cambridge/IBM wrote:

> Issue #91 on the issues list at
> http://www.w3.org/QA/WG/qawg-issues-html
> questions Checkpoint 8.4 of the Spec Guidelines (SpecGL). That
> checkpoint says:
>
> Checkpoint 8.4. Include a statement regarding consistent handling of a
> discretionary item within an implementation. [Priority 2]
> The effect of each individual discretionary item should be consistent
> within a single implementation. For example, a browser's rendering of a
> XSL-FO (XSL Formatting Object) should be the same for every invocation
> regardless of the document instance.
>
> I think the checkpoint needs rewriting. The thrust of it is that any
> individual discretionary item is one that the implementer must decide
> once for the entire implementation. If the spec allows the choice of
> behaviors A or B (but no others), then an implementation must not make
> the product sometimes do A and sometimes B, depending on other factors.

And why is that? The spec authors decided that it is OK to do A *or*
B; why should implementation be constrained beyond that?

> The guideline is worthwhile, but the explanatory matter should point
> out why: so that a test case can be tied to a particular choice.

You can, of course, tie a test case to a particular choice. It is just
that in some cases, a test case will fail to produce a
"success/violation" result, which is very normal for any decent test
suite that works with complex specs. Whether to test "A or B" as two
test cases or one depends on many factors beyond the scope of this
discussion.

IMO, you are putting the carriage in front of the horse. It should be
possible to test that an "A or B" requirement is satisfied. Demanding
that the outcome of the test (outcome-A or outcome-B) is known in
advance and/or is reproduceable adds little value. Compliant
implementations are more important than fewer lines of code in test
suites. Testability of each requirement is the most we should demand,
IMO.

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Monday, 7 October 2002 17:36:30 UTC