Re: OpsGL QA-commitment-group

On Fri, 9 May 2003, Mark Skall wrote:

> I object.  The reason is that I don't accept Alex's premise.  Every
> requirement should (MUST) be testable. (In fact, I thought this
> statement was included somewhere in our guidelines) If a requirement
> is not testable, it should be reworded to be testable or be
> eliminated from the specification.  If it can't be tested, it can't
> be verified that it was done correctly and is, thus, of no use.
> Adding the suggested qualifier would sanction having non-testable
> requirements.

IMO, you are putting the carriage ahead of the horse. The ultimate
goal of every specification is to facilitate compliant
implementations, NOT testable implementations. Of course, we do want
implementations to be testable because it helps us to ensure
compliance, but it is often the case that an implementation is
compliant but not fully testable. In other words, compliance is the
goal, and testability is just one possible way of getting there.

Requirements SHOULD be testable (not "MUST be testable"). Real
protocol specs are full of perfectly valid requirements that can be
implemented correctly, but cannot be tested. Here are two specific
(but very different) examples:

"Every requirement SHOULD be testable"
	This requirement is not testable because there is no
	pragmatic way to test for testability (in general).
	Yet, it is a perfectly valid checkpoint for SpecGL.

"Implementations MUST ignore extension headers they do not understand"
	This requirement is not testable using black-box
	techniques because it is impossible to know that
	something was ignored by a black box (in general).
	Yet, it is a useful requirement that is possible
	(and often easy) to implement correctly.

Here is a list of currently untestable proxy-related MUSTs in HTTP/1.1
(RFC 2616). You will see that many (not all!) of the listed
requirements are not testable at all, cannot be rephrased to become
testable, but it is still possible to implement them correctly.
http://coad.measurement-factory.com/cgi-bin/coad/GraseInfoCgi?info_id=test_group/any/requirement-testable-not

Until all specs and software is written using formal and verifiable
languages, we have to do what the real world does: presume innocent
until proven guilty. If you cannot prove an implementation guilty, you
cannot claim it is not compliant just because some requirements are
not testable. You have to assume it is compliant (for all practical
purposes anyway). Since compliance is the ultimate goal, testability
becomes a SHOULD not a MUST. (Of course, as recent real world events
demonstrate, it is often convenient to presume guilt in order to
conduct the "tests" on weak subjects, but that loophole does not help
in software world because if something is not testable, our
presumptions will not change that status).

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance

Received on Friday, 9 May 2003 11:44:03 UTC