Re: Test suites and RFC2119

On Mon, Jul 4, 2011 at 5:47 AM, Rich Tibbett <richt@opera.com> wrote:
> We currently define tests in test suites for SHOULD requirements. A problem
> occurs when those tests are used to gauge the overall compliance of an
> implementation to the full test suite. An implementation could theoretically
> be 100% compliant without needing to pass non-MUST and non-MUST NOT tests.
>
> Perhaps we should introduce 'bonus' points for SHOULD/SHOULD NOT/MAY and
> RECOMMENDED tests and not have them contribute to overall compliance output,
> thereby allowing implementations to claim 100% compliance to MUST/MUST NOT
> tests. An implementation can then optionally collect any available, optional
> bonus points as available from requirements marked up with other keywords.
>
> Wondering if there is any set W3C thinking on this or a way of including
> SHOULD tests in test suites but clearly indicating that they are, basically,
> optional and do not count towards the overall compliance score? I couldn't
> find anything in [1].

The way HTML5 does it is define different conformance classes:

http://www.whatwg.org/specs/web-apps/current-work/multipage/infrastructure.html#conformance-classes

For instance, some requirements apply only to user agents that support
scripting, some only to user agents that support the default
rendering, etc.  In practice this means that there are few testable
"should" requirements.

Generally, if something is important enough for interop that we want
to test it, we don't want to make it a "should" requirement.  It
should be a "must".  What examples do you have of "should"
requirements that you want to test?

Received on Tuesday, 5 July 2011 22:33:29 UTC