[Testing] Feature testing philosophy

So.... Robert raised an interesting point about complex assertions.  I have
worked out examples to demonstrate how to support said complexity, and I
will post about that separately.  But....  We shouldn't be creating tests
that are that complex.

Let's remember the primary reason for having these tests.  We need to exit
CR.  That means demonstrating that there are at least two implementations
of each feature. Which means we need at least one test per feature.  Let me
say that again in bold: *We need at least one test per feature.*

Take the body and bodyValue properties.

We need a body test.  That test needs to check the various aspects of body.

We need a SEPARATE bodyValue test.  That test needs to check the aspects of
bodyValue.

If there is a requirement that there be no bodyValue property when body is
present in an annotation, then that is part of the body test.  No problem.

But we cannot be combining these two tests. It is the wrong level of
granularity.  We need a result PER FEATURE PER IMPLEMENTATION.

So - let's concentrate on that level of granularity.  If, later, we also
want to create some sort of super schema that tests everything so that you
can validate the shape of any given annotation... that's a cool exercise,
but it is not part of this project.

At least, that's my opinion.

-- 
Shane McCarron
Projects Manager, Spec-Ops

Received on Thursday, 14 July 2016 15:17:28 UTC