CR Testing rules for Annotation Data Model

TL;DR: I am working on stubs and thinking about rules for writing and
combining tests.  This email proposes some rules.  Feel free to disagree,
suggest modifications, or proclaim your support ;-)

I have been looking at the annotation-model spec again through the lens of
"what do we need to have tests for to evaluate features".  This raises a
general question that I thought it would be good for experienced testing
people to chime in on.  Basically, that is "what constitutes a feature in a
W3C spec? And when is a feature testable?"

Good news! W3C has guidance on this at [1].  Bad news! It isn't super
helpful.  The W3C also has a document about writing tests [2].  This
document should actually be required reading for everyone who works on a
W3C spec.  Alas, it is likely never seen by most.  I won't ask you all to
read it.  It suggests a lot of good ways for evaluating testable
assertions.  It also has some recommendations for marking up the spec and
test cases so that they are comparable.  Also alas, we have done none of
this!

So.... what should we do?  Here's my suggestion.  We're smart people, and
this is not a tough problem.  Some rules:

   1. If there is a MUST or a MUST NOT in the spec and it applies to the
   client, have a test for it.  Do these FIRST.
   2. If there is a SHOULD or a SHOULD NOT in the spec and it applies to
   the client, do these NEXT.  (If there is a should that can be combined in a
   test with a MUST in a way that makes sense, go for it - but remember that
   since it is a should the result does not effect the pass or fail of the
   test - it is just informative).
   3. If there is a MAY in the spec and it applies to the client, ONLY do
   these if they are a conditional feature that has a MUST or SHOULD
   requirement (e.g., "A client MAY support this insane thing.  If it does so,
   it MUST do it in this way.")  Hopefully there are none of these.

As to writing the tests themselves, here are some guidelines for creating
and grouping assertions:

   - If a test evaluates multiple assertions, the WORST result report of
   all the assertions is the result of the test.
   - It is possible to have multiple separate tests within a test case.  If
   they are discrete "tests" within WPT they will each have a result.  The
   Annotation Model framework I have developed supports this if you list a
   bunch of top level assertions.  However, nested assertions are treated as
   "sub-tests" and their result rolls up to the top level and the worst result
   is used.  This happens automatically.
   - If there is a MUST or a MUST NOT then it is a conformance
   requirement.  The evaluation of that condition needs to be reflected in the
   result of running the test that evaluates that requirement.  Therefore,
   there really needs to be an individual top level assertion for each such
   requirement. Example: @context property.
   - If there is a MAY, SHOULD, or a SHOULD NOT that is part of a
   "condition" then it is also a conformance requirement.  As such it needs to
   be treated as a MUST or MUST NOT for purposes of evaluation whether a
   feature is supported or not. Example: body and bodyValue properties.
   - If there is a MAY, SHOULD, or a SHOULD NOT that is not part of a
   "condition", it is NOT a conformance requirement.  We can evaluate these if
   we like, but the evaluation must not cause the evaluation of a MUST or a
   MUST NOT requirement to fail.  Example: motivation property.
   - If it makes sense to do so with the same supplied input, these
   assertions can all be contained in a single ".test" file.

Looking at the spreadsheet Tim has created [3] I see that there are only 7
MUST or MUST NOT requirements.  Does that mean there are only 7 tests that
we need to write?  Wouldn't that be nice?  No.  It means that there are a
bunch of assertions in the spec that are "conditional". A quick scan seems
to imply there are quite a few of these.  I am going to do a deep dive to
try to determine the extent as I create the stubs as per my action item
from Friday.



[1] https://w3c.github.io/testing-how-to/
[2] https://www.w3.org/TR/test-methodology/

-- 
Shane McCarron
Projects Manager, Spec-Ops