Re: Test suites and RFC2119

It's best to view RFC 2119 in the context of IETF rules for interoperability:

Progression along standards track depends on there being multiple independent interoperable implementations of every feature.

While "feature" is not clearly defined, I believe that in a well-written specification, any normative language (MUST, MAY, SHOULD) is associated with a feature.

Two implementations "interoperate" and are "interoperable implementations of a feature" if they both implement the feature and "interoperate".
"interoperate" is interpreted slightly differently by the context, but if you think a web page is an implementation and a browser is an implementation, then they "interoperate" if the browser does what the author of the web page expected.

"MAY" features are not required to be implemented, but if they are implemented, they should operate as described in the spec and the interoperable implementations treat the feature as expected.

"MUST" features are required to be implemented.  A well-written spec only mandates "MUST" in cases where it is required for interoperability.

For test suites, it seems best to treat 'SHOULD' as if it were 'MUST but you can apply for a waiver'.

That is, test cases should test for SHOULD, but there are situations or contexts where interoperability is accomplished some other way... and that the way to test for "SHOULD" is to test as if it were "MUST", but let the implementor provide a convincing proof of why not following the normative language does not interfere with interoperability.


Larry
--
http://larry.masinter.net