Re: Should we test SHOULD?

#3 sounds like the best option to me, with a simple toggle for whether the
SHOULD tests are run. Perhaps something like the Skip JSHint option in the
jQuery UI test suites:
http://view.jqueryui.com/master/tests/unit/progressbar/progressbar.html

Would the SHOULD tests run by default or would you need to toggle them on?


On Sun, Sep 22, 2013 at 5:26 AM, Tobie Langel <tobie@w3.org> wrote:

> Hi all,
>
> A pull request[1] on the main repo brought up the issue of how to handle
> optional normative requirements (SHOULD, MAY, etc.). It's not the first
> time this issue was discussed. I found, for example, a rather long thread
> on this topic[2] on the public-webapps mailing list. I couldn't find,
> however a recommended practice on how we should handle this. I'd like us to
> agree on one and document it.
>
> Here's a number of propositions:
>
> 1) We only test MUST normative requirements.
>
> 2) We test all normative requirements, and rely on result interpretation
> to determine whether an implementation conforms to the spec (an
> implementation can fully conform even though it fails a number tests, as
> long as those are determined to be SHOULD/MAY tests).
>
> 3) We test all normative requirements but add meta data to those tests
> that aren't MUST requirements. This allows running subset of tests when
> SHOULD requirements don't make sense. E.g. avoid running media capture
> tests on a device that doesn't have a camera.
>
> I'd be inclined to go with 3), but I'm eager to hear other's thoughts on
> the subject.
>
> Thanks,
>
> --tobie
> ---
> [1]: https://github.com/w3c/web-platform-tests/pull/306
> [2]:
> http://lists.w3.org/Archives/Public/public-webapps/2011JulSep/0053.html
>
>
>
>