Re: Test suites and RFC2119

On Mon, 04 Jul 2011 11:47:22 +0200, Rich Tibbett <richt@opera.com> wrote:

> RFC2119 'Key words for use in RFCs to Indicate Requirement Levels'  
> defines the keyword 'SHOULD' as:
>
> "This word, or the adjective "RECOMMENDED", mean that there
>   may exist valid reasons in particular circumstances to ignore a
>   particular item, but the full implications must be understood and
>   carefully weighed before choosing a different course."
>
> Generally, I think we can agree that anything less than MUST or MUST NOT  
> requirements in a spec are pretty useless when it comes to conformance  
> testing. We try to write specs to these keywords but other keywords tend  
> to creep in to most specifications.

Hmm. No, I think a spec that uses these correctly has requirements where  
there are legitimate reasons not to do the usual thing, but a strong case  
for doing it unless there is a clear reason.

> We currently define tests in test suites for SHOULD requirements. A  
> problem occurs when those tests are used to gauge the overall compliance  
> of an implementation to the full test suite. An implementation could  
> theoretically be 100% compliant without needing to pass non-MUST and  
> non-MUST NOT tests.

I don't think so. RFC 2119 is pretty clear about what must and should  
mean. As Bjoern said a numerical score for tests is generally not that  
useful. But compliance is doing all the things you must, and in practical  
terms that can readily be translated to *at least* passing all the test  
that test things the spec says must happen.

> Perhaps we should introduce 'bonus' points for SHOULD/SHOULD NOT/MAY and  
> RECOMMENDED tests and not have them contribute to overall compliance  
> output, thereby allowing implementations to claim 100% compliance to  
> MUST/MUST NOT tests. An implementation can then optionally collect any  
> available, optional bonus points as available from requirements marked  
> up with other keywords.
>
> Wondering if there is any set W3C thinking on this or a way of including  
> SHOULD tests in test suites but clearly indicating that they are,  
> basically, optional and do not count towards the overall compliance  
> score? I couldn't find anything in [1].

I don't think it's a good idea to do numerical scoring of conformance. Yu  
would need to be sure that the things being tested are of equal  
importance, or work out how to balance the scores. And the cost of that  
already outweighs the benefit by so much it seems pointless to go there.

just my 2 cents

chaals

> - Rich
>
> [1] http://www.w3.org/TR/test-methodology/


-- 
Charles McCathieNevile  Opera Software, Standards Group
     je parle français -- hablo español -- jeg lærer norsk
http://my.opera.com/chaals       Try Opera: http://www.opera.com

Received on Monday, 4 July 2011 17:09:59 UTC