Re: Test suites and RFC2119

On Sun, 10 Jul 2011 22:34:59 +0200, Aryeh Gregor  
<Simetrical+w3c@gmail.com> wrote:

> On Sun, Jul 10, 2011 at 3:59 PM, Charles McCathieNevile
> <chaals@opera.com> wrote:
>> Privacy and security restrictions leap to mind. There are things that  
>> really are "should" requirements because there are valid use cases for
>> not applying them, and no reason to break those cases by making the
>> requirement a "must". In the normal case where they are applied you
>> want to be able to test.
>
> Were you thinking of specific examples?  I can't think of any offhand.

The most recent example is the Contacts API from the DAP group, but this  
is a pretty common pattern. Geolocation went through a long discussion on  
the topic.

>> But the difference between "should" and "must" is already two sets of
>> conformance profiles (or whatever you want to call it), where one  
>> applies always and the other applies unless there's a reason not to
>> do the thing that is assumed to be normal.
>
> The difference is that if you have "must" requirements that are
> specific to a single conformance class, you can write a test suite and
> expect every implementation in that class to pass it.

Yep.

> For "should" requirements, you're saying it's okay to violate it, so
> test suites don't make a lot of sense.

Not quite. I'm saying that there are cases where violoating the  
requirement is reasonable, so test results shouldn't determine simple  
conformance.

On the other hand, where these are things that in *most* cases we want  
interoperability, it makes sense to have test suites so people who don't  
violate the requirement can check that what they are doing is consistent  
with what others do.

Essentially the question shouldn't revolve around the tests themselves -  
tests can be useful even when they're not determining "pure conformance".  
What matters is how you use the results.

Performance tests are another class of tests that can be very useful,  
allowing people to understand what sort of things work well and what sort  
of things might cause real-world problems, without normally giving any  
information about actual conformance.

cheers

-- 
Charles McCathieNevile  Opera Software, Standards Group
     je parle français -- hablo español -- jeg lærer norsk
http://my.opera.com/chaals       Try Opera: http://www.opera.com

Received on Sunday, 10 July 2011 21:05:34 UTC