Re: [Testing] Feature testing philosophy

I feel like we reached agreement that this approach is the logical one for
getting the initial collection of tests.  As I mentioned, my goal is to
have that initial collection in a form that is manually operable.  Once
that is done (or at least getting going) I will do a proof of concept
automated test using WebDriver to populate the test interface and click
"go".  Feeding data in from a collection of annotation samples that we
already have from the spec itself.  Such a PoC could be used by actual
implementors as a model and they could push data in from their actual
implementations instead of using the stock sample data.

Thanks everyone for the discussion today.  I think it was really productive.

On Thu, Jul 14, 2016 at 10:16 AM, Shane McCarron <shane@spec-ops.io> wrote:

> So.... Robert raised an interesting point about complex assertions.  I
> have worked out examples to demonstrate how to support said complexity, and
> I will post about that separately.  But....  We shouldn't be creating tests
> that are that complex.
>
> Let's remember the primary reason for having these tests.  We need to exit
> CR.  That means demonstrating that there are at least two implementations
> of each feature. Which means we need at least one test per feature.  Let me
> say that again in bold: *We need at least one test per feature.*
>
> Take the body and bodyValue properties.
>
> We need a body test.  That test needs to check the various aspects of body.
>
> We need a SEPARATE bodyValue test.  That test needs to check the aspects
> of bodyValue.
>
> If there is a requirement that there be no bodyValue property when body is
> present in an annotation, then that is part of the body test.  No problem.
>
> But we cannot be combining these two tests. It is the wrong level of
> granularity.  We need a result PER FEATURE PER IMPLEMENTATION.
>
> So - let's concentrate on that level of granularity.  If, later, we also
> want to create some sort of super schema that tests everything so that you
> can validate the shape of any given annotation... that's a cool exercise,
> but it is not part of this project.
>
> At least, that's my opinion.
>
> --
> Shane McCarron
> Projects Manager, Spec-Ops
>



-- 
Shane McCarron
Projects Manager, Spec-Ops