Re: Test suites and RFC2119

On Mon, Jul 4, 2011 at 10:47 AM, Rich Tibbett <richt@opera.com> wrote:
> RFC2119 'Key words for use in RFCs to Indicate Requirement Levels' defines
> the keyword 'SHOULD' as:
>
> "This word, or the adjective "RECOMMENDED", mean that there
>  may exist valid reasons in particular circumstances to ignore a
>  particular item, but the full implications must be understood and
>  carefully weighed before choosing a different course."
>
> Generally, I think we can agree that anything less than MUST or MUST NOT
> requirements in a spec are pretty useless when it comes to conformance
> testing. We try to write specs to these keywords but other keywords tend to
> creep in to most specifications.
>
> We currently define tests in test suites for SHOULD requirements. A problem
> occurs when those tests are used to gauge the overall compliance of an
> implementation to the full test suite. An implementation could theoretically
> be 100% compliant without needing to pass non-MUST and non-MUST NOT tests.

You can usually make those disappear. Or you can use the word OPTIONAL
and make the conformance requirement open ended (thus making it
untestable)... so, instead of "X SHOULD support Y", you can have "It
is OPTIONAL for X to support other technologies (e.g., Y)".  The
weasels your way out of a testing some things :)

> Perhaps we should introduce 'bonus' points for SHOULD/SHOULD NOT/MAY and
> RECOMMENDED tests and not have them contribute to overall compliance output,
> thereby allowing implementations to claim 100% compliance to MUST/MUST NOT
> tests. An implementation can then optionally collect any available, optional
> bonus points as available from requirements marked up with other keywords.
>
> Wondering if there is any set W3C thinking on this or a way of including
> SHOULD tests in test suites but clearly indicating that they are, basically,
> optional and do not count towards the overall compliance score? I couldn't
> find anything in [1].

I was intending to modify the test suite and implementation reports
template to do that. Some tests in Dig Sig and P&C are already marked
as type="optional".

At the end of the day, it is people that evaluate conformance and
ratify specs... so it's not really a technical issue.

> [1] http://www.w3.org/TR/test-methodology/

[1] needs so love. Will update it soon.



-- 
Marcos Caceres
http://datadriven.com.au

Received on Monday, 4 July 2011 10:56:16 UTC