RE: [Testing] Feature testing philosophy

So currently the bodyResource.json [1] schema in w3c/web-annotation-tests conflates together checks to see if the annotation body is correct. An annotation passes this schema if the body is any of textualBody, choice or set, specific resource, external web resource (string of format uri), external web resource with additional properties (json object with at least an id key that is a string of uri format). If the body is an array, it checks that at least one of the items matches one of the definitions above.

If I understand what we need is a separate schema for each of the possible body classes in order to report which body class(es) -- i.e., features -- have been implemented in a given annotation. Easy to do since the definitions have been written this way, but just trying to understand how these schemas will be used by the test scripts.  So, we end up with:

Schema 1: test for bodyValue
Schema 2: test for body that is a TextualBody, or for body that is an array containing a TextualBody
Schema 3: test for body that is a SpecificResource, or for body that is an array containing a SpecificResource
Schema 4: test for body that is a Choice, or for body that is an array containing a Choice object
Schema 5: test for body that is a Set, or for body that is an array containing a Set
Schema 6: test for body that is an ExternalWebResource, or for body that is an array containing an ExternalWebResource

Is this the direction we're heading now?

[1] https://github.com/w3c/web-annotation-tests/blob/master/annotations/bodyResource.json

Thanks,
Tim Cole



________________________________
From: Shane McCarron [shane@spec-ops.io]
Sent: Friday, July 15, 2016 11:55
To: W3C Public Annotation List
Subject: Re: [Testing] Feature testing philosophy

I feel like we reached agreement that this approach is the logical one for getting the initial collection of tests.  As I mentioned, my goal is to have that initial collection in a form that is manually operable.  Once that is done (or at least getting going) I will do a proof of concept automated test using WebDriver to populate the test interface and click "go".  Feeding data in from a collection of annotation samples that we already have from the spec itself.  Such a PoC could be used by actual implementors as a model and they could push data in from their actual implementations instead of using the stock sample data.

Thanks everyone for the discussion today.  I think it was really productive.

On Thu, Jul 14, 2016 at 10:16 AM, Shane McCarron <shane@spec-ops.io<mailto:shane@spec-ops.io>> wrote:
So.... Robert raised an interesting point about complex assertions.  I have worked out examples to demonstrate how to support said complexity, and I will post about that separately.  But....  We shouldn't be creating tests that are that complex.

Let's remember the primary reason for having these tests.  We need to exit CR.  That means demonstrating that there are at least two implementations of each feature. Which means we need at least one test per feature.  Let me say that again in bold: We need at least one test per feature.

Take the body and bodyValue properties.

We need a body test.  That test needs to check the various aspects of body.

We need a SEPARATE bodyValue test.  That test needs to check the aspects of bodyValue.

If there is a requirement that there be no bodyValue property when body is present in an annotation, then that is part of the body test.  No problem.

But we cannot be combining these two tests. It is the wrong level of granularity.  We need a result PER FEATURE PER IMPLEMENTATION.

So - let's concentrate on that level of granularity.  If, later, we also want to create some sort of super schema that tests everything so that you can validate the shape of any given annotation... that's a cool exercise, but it is not part of this project.

At least, that's my opinion.

--
Shane McCarron
Projects Manager, Spec-Ops



--
Shane McCarron
Projects Manager, Spec-Ops