Re: Test suites and RFC2119

On Sun, Jul 10, 2011 at 5:04 PM, Charles McCathieNevile
<chaals@opera.com> wrote:
> Not quite. I'm saying that there are cases where violoating the requirement
> is reasonable, so test results shouldn't determine simple conformance.
>
> On the other hand, where these are things that in *most* cases we want
> interoperability, it makes sense to have test suites so people who don't
> violate the requirement can check that what they are doing is consistent
> with what others do.

More specifically, we should have a set of requirements that we expect
all major browsers to follow out of the box, and a full test suite
associated with those requirements.  (I assume browser-targeted
standards here, for the sake of argument.)  That way other
implementations can verify that, in practice, they interoperate with
all the major browsers.  If we expect some implementations to only
implement parts of the specification, it's useful to separate the
requirements into classes, so that (for instance) a non-interactive
HTML5 processor can verify that it's parsing HTML the same as major
browsers, without being distracted by the noise about all the
interaction-related requirements it fails.

On the other hand, if you use "should" for requirements that all major
browsers intend to conform to, but also for things that they don't,
you reduce the usefulness of any "should"-related test suite.  Some of
the tests are things that any implementation wants to pass if it
intends to be fully compatible, and some are things that they can
ignore in practice.  Separating these into distinct test suites is
valuable, and that's what "must" requirements with multiple
conformance classes permits.

On Sun, Jul 10, 2011 at 6:17 PM, Bjoern Hoehrmann <derhoermi@gmx.net> wrote:
> And if I make an implementation that does not fit in any of the classes
> I can just argue that the specification did not anticipate the class my
> implementation falls in. You would have to explain how arguing about a
> missing conformance class is better than arguing about whether "should"
> level requirements have been met. With your model you would have more
> "clarity", but you would also be more wrong, and require more effort to
> make things right, in addition to inhibiting innovation. I think that's
> a very difficult argument to make and we have "should" because of that.

The standards we're discussing are not coercive.  They and their
associated test suites exist solely to assist interested parties in
writing software that will operate interoperably.  To that end,
conformance requirements and test suites should be designed so as to
best permit interested implementers to write interoperable
implementations.  The point isn't whether you can argue that you do or
don't conform to the letter of the standard, it's whether your
implementation is interoperable in practice, and that's the only thing
tests should be targeted at.  Having a suite of "should" tests that
mix together things we expect content might depend on (because major
browsers agree) and things we don't think it will depend on (because
they don't agree, or the difference isn't visible to content) is less
useful than dividing up test suites based on interoperability impact.

I don't foresee any major arguments here on the specifics.  It's
usually pretty clear whether a given requirement affects
interoperability or not.  If it does, and it looks like we can get the
major implementations to agree, make it "must".  If it doesn't, or we
can't, make it "should" or "may".  How to split the "must"
requirements up into conformance classes is unlikely to be
particularly controversial or hard to decide -- I'm not aware of any
significant disagreement that's come up in HTML5.  There aren't going
to be that many classes of UAs in practice, even for a huge standard
like HTML5.

So I continue to feel that if a requirement has potential
interoperability implications and we can get browsers to agree on it,
it should be a "must".  This covers practically all requirements that
are readily testable anyway, so it would leave little need to consider
a "should" test suite.

Received on Monday, 11 July 2011 16:59:00 UTC