Re: Should we test SHOULD?

On Sep 24, 2013, at 3:03 AM, Tobie Langel wrote:

> On Monday, September 23, 2013 at 12:31 PM, Robin Berjon wrote:
>> On 23/09/2013 11:48 , James Graham wrote:
>>> Therefore I conclude that we should take option 1); simply consider
>>> "should" level conditions in the spec as untestable (for us) as other
>>> implementation-specific requirements on UI. If you really want to call
>>> out these somehow, I would be happy for people to write should.txt files
>>> describing all the should level conditions to guide people writing
>>> implementation-specific tests.
>> 
>> I would opt for a "modified (1)". By default, don't test SHOULD (they 
>> normally aren't testable) but allow people to use their better judgement 
>> and include tests for SHOULD in cases where they feel the specification 
>> was exceedingly cautious.
>> 
>> The issue of devices not being able to support some functionality is 
>> orthogonal, and should really not be handled by use of SHOULD in 
>> specifications. For instance, an eInk device won't render colour, but it 
>> would not IMHO be a good use of anyone's time to put all mention of 
>> colour behind SHOULD in any CSS module that relates to it.
> 
> It would be interesting to hear the CSS WG's perspective on the matter. Crude stats over the csswg-test repository report the following:
> 
>    $ grep "flags" -r ./ | grep "should" | wc -l
>    28
>    $ grep "flags" -r ./ | grep "may" | wc -l
>    214
> 
> 
> 
> i.e. there's roughly 30 tests that are marked as testing SHOULD and over 200 that are marked as testing MAY.
> 
> Peter, Rebecca, fantasai, others, any thoughts on how to best handle SHOULD/MAY requirements in specs? For ref, the thread starts here:
> 
> http://lists.w3.org/Archives/Public/public-test-infra/2013JulSep/0240.html 

The CSSWG (obviously) does test for SHOULD and MAY requirements and we use your option 3 (metadata in the test to identify).

While the use of SHOULD/MAY varies in sometimes subtle ways, often it reads as, "you don't have to do this, but if you do, this is how it must be done". Ideally, in those circumstances, it should also be clear in the spec in question what the behavior must be if the feature is not implemented. The ideal is that tests for those kinds of requirements should pass if the requirement is implemented per the spec, fail if it's implemented in a manner not compatible with the spec, and have a neutral result, such as "n/a" if the feature is not implemented.

The CSSWG test harness also takes the SHOULD/MAY flags into account when determining if a spec meets CR exit criteria (in that it counts them separately and considers a spec to have met it's CR exit criteria if it has multiple passes from all non-SHOULD/MAY tests). 

While SHOULD/MAY tests don't add any value for CR exit testing, they can be useful for conformance testing. They can also be useful by gathering implementation data when developing future levels of a spec where turning a SHOULD/MAY into a MUST is being considered. 

Peter