RE: Should we test SHOULD?

> From: Tobie Langel [mailto:tobie@w3.org]
> Sent: Sunday, September 22, 2013 5:26 PM
> To: public-test-infra
> Subject: Should we test SHOULD?
> 
> Hi all,
> 
> A pull request[1] on the main repo brought up the issue of how to handle
> optional normative requirements (SHOULD, MAY, etc.). It's not the first time
> this issue was discussed. I found, for example, a rather long thread on this
> topic[2] on the public-webapps mailing list. I couldn't find, however a
> recommended practice on how we should handle this. I'd like us to agree on
> one and document it.
> 
> Here's a number of propositions:
> 
> 1) We only test MUST normative requirements.
> 
> 2) We test all normative requirements, and rely on result interpretation to
> determine whether an implementation conforms to the spec (an
> implementation can fully conform even though it fails a number tests, as long
> as those are determined to be SHOULD/MAY tests).
> 
> 3) We test all normative requirements but add meta data to those tests that
> aren't MUST requirements. This allows running subset of tests when SHOULD
> requirements don't make sense. E.g. avoid running media capture tests on a
> device that doesn't have a camera.

3) looks good to me.

In [3], CSS test defines 2 flags token 'should' and 'may' in
<meta name="flags" content="TOKENS" />, with description as

may:  Behavior tested is preferred but OPTIONAL. [RFC2119] (These tests must be reviewed by a test suite owner or peer.)
should:  Behavior tested is RECOMMENDED, but not REQUIRED. [RFC2119]

I think we can leverage this.

> 
> I'd be inclined to go with 3), but I'm eager to hear other's thoughts on the
> subject.
> 
> Thanks,
> 
> --tobie
> ---
> [1]: https://github.com/w3c/web-platform-tests/pull/306

> [2]: http://lists.w3.org/Archives/Public/public-

> webapps/2011JulSep/0053.html
> 
[3] http://wiki.csswg.org/test/format#requirement-flags

Received on Monday, 23 September 2013 01:02:14 UTC