Test suites and RFC2119

RFC2119 'Key words for use in RFCs to Indicate Requirement Levels' 
defines the keyword 'SHOULD' as:

"This word, or the adjective "RECOMMENDED", mean that there
  may exist valid reasons in particular circumstances to ignore a
  particular item, but the full implications must be understood and
  carefully weighed before choosing a different course."

Generally, I think we can agree that anything less than MUST or MUST NOT 
requirements in a spec are pretty useless when it comes to conformance 
testing. We try to write specs to these keywords but other keywords tend 
to creep in to most specifications.

We currently define tests in test suites for SHOULD requirements. A 
problem occurs when those tests are used to gauge the overall compliance 
of an implementation to the full test suite. An implementation could 
theoretically be 100% compliant without needing to pass non-MUST and 
non-MUST NOT tests.

Perhaps we should introduce 'bonus' points for SHOULD/SHOULD NOT/MAY and 
RECOMMENDED tests and not have them contribute to overall compliance 
output, thereby allowing implementations to claim 100% compliance to 
MUST/MUST NOT tests. An implementation can then optionally collect any 
available, optional bonus points as available from requirements marked 
up with other keywords.

Wondering if there is any set W3C thinking on this or a way of including 
SHOULD tests in test suites but clearly indicating that they are, 
basically, optional and do not count towards the overall compliance 
score? I couldn't find anything in [1].

- Rich

[1] http://www.w3.org/TR/test-methodology/

Received on Monday, 4 July 2011 09:47:50 UTC