RE: placeholder test files

Shane-
 
This helps a lot – not as far apart as I feared. 
 
A passAndMessageAndContinue option would be nice.  But I think we may still need to explore test granularity and the use of failAndSkip a little more. 
We may need a notFoundAndSkip option (as described below). 
 
So, my understanding is that a tester pastes an annotation into a dialog in her Web browser and then presses test.  Before doing so either by what directory she enters on the form or in some other way she could narrow or select which tests are to be run.  Is that how it would work? Do all the web-annotation tests run each time or is it better to subset? 
 
So, for example, If we have a test labeled EmbeddedTextualBody that is to be run, it's presumably only meaningful for an annotation that has at least one EmbeddedTextualBody, e.g., see the difference between Example 2 and Example 5 from the data model below.  I was assuming that when we got to the point in this EmbeddedTextualBody test of checking Example 2 and discovering that the body object does not have a value key (and therefore is not an EmbeddedTextualBody), we would not test further for other keys as part of the EmbeddedTextualBody test. In other words we would want to report not found and skip the rest of the test. 
 
It's not that there's anything wrong with Example 2 as an annotation (so from what you said, we don't want to fail it, I guess), but I presumably will not want to report that it implements the use (for example) of language on an EmbeddedTextualBody since that's not true strictly speaking. I could of course report simply that language key was implemented, but is this enough?   
 
This basically comes down to a question of whether I should report implementation of a key independent of its context. If implementerA implements language on an ExternalWebResource do I need to differentiate this result from when implementerB implements language on an EmbeddedTextualBody?  
 
In many cases this distinction may not be significant, but in some cases it could be. For example, for illustration, I've added an id to example 5 (EmbeddedTextualBody). In an annotation, the id key can appear on both ExternalWebResource objects and on EmbeddedTextualBody objects, but for ExternalWebResource objects the id key is a MUST. For EmbeddedTextualBody objects, the id key is a MAY.  So it seems to me I need a way to not test for id key when a particular class of body or target is not found in order to avoid confusion in my reporting.  I may not want to report a failure in such cases, but I need to skip a set of tests that don't apply to avoid (for example) failing an EmbeddedTextualBody that doesn't have an id, or avoid passing a ExternalWebResource value that is lacking an id. 
 
There are various ways to do this. I was using failAndSkip, but maybe notFoundAndSkip (or the like) would be better.  What do you think?
 
Example 2: External Web Resources
{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "id": "http://example.org/anno2",
  "type": "Annotation",
  "body": {
    "id": "http://example.org/analysis1.mp3",
    "format": "audio/mpeg",
    "language": "fr"
  },
  "target": {
    "id": "http://example.gov/patent1.pdf",
    "format": "application/pdf",
    "language": ["en", "ar"],
    "textDirection": "ltr",
    "processingLanguage": "en"
  }
}
 
Example 5: Textual Body
{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "id": "http://example.org/anno5",
  "type":"Annotation",
  "body": {
    "id" : "http://example.org/myTextualBody1.html",
    "type" : "TextualBody",
    "value" : "<p>j'adore !</p>",
    "format" : "text/html",
    "language" : "fr"
  },
  "target": "http://example.org/photo1"
}
 
-Tim Cole
 
From: Shane McCarron [mailto:shane@spec-ops.io] 
Sent: Thursday, July 21, 2016 10:56 AM
To: t-cole3 <t-cole3@illinois.edu>
Cc: W3C Public Annotation List <public-annotation@w3.org>; Discussions about Test Development <testdev@lists.spec-ops.io>
Subject: Re: placeholder test files
 
Thanks for your detailed email.  I have put detailed comments inline.  Let me preface everything by saying that this is almost all philosophy of testing stuff.  There are indeed may ways to skin this cat.  None of them are really wrong. And I agree that we should do everything in a consistent manner to the extent that makes sense.  Having said that...
 
On Wed, Jul 20, 2016 at 4:37 PM, Timothy Cole <t-cole3@illinois.edu <mailto:t-cole3@illinois.edu> > wrote:
Shane-
 
I'm confused again, especially about the distinction between assertion files / schemas (*.json) and tests (*.test) and how we want to use them – realizing that there are multiple ways to skin a cat, but wanting to make sure we settle collectively on the same way. We're probably in the same ball park, but your examples make me think I heading in the opposite direction from you.
 
So, Jacob, Janina, and I were going forward on the assumption that we needed individual assertion files (json schemas) for each of the properties defined in the Web Annotation model. Tests would reference these assertion files rather than embed schemas directly, and we would use the "onUnexpectedResult" : "failAndSkip", "onUnexpectedResult" : "failAndContinue", "onUnexpectedResult" : "succeedAndSkip", "onUnexpectedResult" : "suceedAndContinue" to manage the flow of assertions checked and would report error messages, etc.   The advantage of not embedding the schemas in the tests was that same keys are used on multiple objects defined in the model and so reusability should be facilitated. 
 
Sure.  That's a fine way to go.   I will get into some detail below, but in general I would say that external schema are preferred in any case where there is possible reuse. I was putting them in line in my examples because it makes the example easier to comprehend for some.  
 
As to the flow of assertions and running tests... in general I believe it is a good idea to evaluate all of the assertions in a test unless doing so would result in catastrophe (e.g., testing laser firing controls and discovering that the off control is disabled should result in an abort because you would start firing the laser and burn down the building.  This actually happened to me once.)  So in the case of our tests, when evaluating any message, go ahead and test all of the bits that might be in a message of a given shape.  If some piece is mandatory and missing or wrong, fail the test.  If some piece is optional and missing, pass.  If some piece is optional and present and wrong, fail.  This is particularly important with regard to reporting.  In order for the reports to generate cleanly and all of the data to line up well, all of the tests need to be run by every platform.  I hope that is clear and I expect we agree fully.
 
More details below:
 
 
 
Toward that end we uploaded json schemas yesterday designed to detect  and validate (to a limited extent) format, language, processingLanguage, textDirection, id. The first four of these (variously should and may) can appear on Textual Bodies, External Web Resource descriptions that appear as Bodies, Targets, SpecificResource Sources, items in Choice, List, Composite, and Independents. id can appear on these and on additional objects. We also uploaded a few schemas that can be used as assertions in a test to detect a Textual Body (using presence of the value key in certain conditions [a must] rather than the type value, since the type value for Textual Bodies is only a Should), as well as schemas designed to recognize the use of Choice, List, Composite or Independents individually (but are working on more schemas for these and for Specific Resources).  There were previously schemas uploaded to check for presence and correctness of @context, id, "type": "Annotation", target(s), etc., though I'm not sure they're in the right place. Our assumption was that other schema files and tests would then be written to leverage these individual assertion files (i.e., schemas), e.g.,
 
I think that is a great approach.  I was working in parallel with you and embedded my assertions in some cases.  But there is no reason to do that.  I would rather reference external schema definitions - in particular so that as those are refined the refinements are reflected everywhere.
 
 
Given a purported Annotation, a test for Embedded Textual Body (if selected) would proceed something like:
 
1. Reference an assertion file (schema) that checked for all annotation MUSTS (referencing the @context, id, … assertion files and providing error messages that enumerate failings agains Annotation MUSTS as appropriate)
     "onUnexpectedResult" : "failAndSkip" – json submitted does not meet MUST requirements to be an annotation, no further testing.
 
Disagree here.  There is no reason to stop.  All of the assertions are embedded in or referenced from the same test case file (.test) so they can be easily run and there is no reason to not run them as there is no cost.
 
2. Reference a schema to detect body(ies) of the Annotation that satisfy the MUST requirement for Embedded Textual body. 
     "onUnexpectedResult" : "failAndSkip" – json submitted does not implement Embedded Textual Body feature of the model, no further testing of Embedded Textual Body features.
 
I think I disagree here as well. Because as I said above, each platform should run the same tests so we can do apples to apples comparisons.  Also because there might be a small error in this part but the rest of the annotation might be okay. Or, more importantly, there might be other errors and if an implementer is fixing things based upon this report, they might as well fix everything all at once.
 
 
3. Check if the Embedded Textual Body has a language property, if yes, then the Annotation's use of Textual Body implements the language feature for that kind of body.
     "onUnexpectedResult" : "failAndContinue" – warn that language (a Should) was not implemented
 
4. Check if the Embedded Textual Body has a format property, if yes, then the Annotation's use of Textual Body implements the format feature.
     "onUnexpectedResult" : "failAndContinue" – warn that format (a Should) was not implemented
 
This is not a conformance failure.  It might be valuable information for the implementer.  But it is not a FAIL if the feature is not present.  It is only a FAIL if the feature is present and implemented wrong.  Otherwise it is a PASS.  I am checking with the WPT gurus about how they normally handle SHOULD assertions in conformance tests.  We need to be consistent with common use I think.
 
 
5. Check if the Embedded Textual Body has a processingLanguage property, if yes, then the Annotation's use of Textual Body implements the processingLanguage feature.
     "onUnexpectedResult" : "failAndContinue" – inform that processingLanguage (a May) was not implemented
 
6. Check if the Embedded Textual Body has a textDirection property, if yes, then the Annotation's use of Textual Body implements the textDirection feature.
     "onUnexpectedResult" : "failAndContinue" – inform that textDirection (a May) was not implemented
 
7. Check if the Embedded Textual Body has a id property, if yes, then the Annotation's use of Textual Body implements the id feature.
     "onUnexpectedResult" : "failAndContinue" – inform that id (a May) was not implemented
 
A MAY was not implemented is not a FAIL.  Unfortunately, there does not seem to be a clear way to send information along with a PASS result.  I am going to dig into that.  If there is a way then we could extend our declarative syntax to include "passAndMessageAndContinue" that adds a message or something. Anyway, I will check into that for SHOULD and MAY assertions - I appreciate what you are trying to do but we can't mark then as FAIL if the test is indeed passing.  That would make our implementation report look like... well, like everything was failing.
 
 
Similar logic for checking for External Web Resources as bodies, but a separate test to see if EWR were implemented as a target. EWRs can also be SpecificResource sources. Textual Bodies and EWRs can be items in a Choice, List, Composite or Independents arrays. 
 
The test logic gets longer for Bodies or Targets that are SpecificResources, Choice, List, Composite or Independents, but the basic flow looks to be similar.
 
Sure - makes sense.
 
 
When body or target is an array of mixed body/target types, this approach can only report on items matching the pattern being checked, i.e., only the Embedded Textual Bodies in an array of mixed Textual Bodies and External Web Resources – so we might run into an issue of reporting that annotation implemented Textual Bodies (i.e., correctly) but also had bodies that did not match any valid bodies. To avoid this, presumably, just as we will check all annotations for Annotation MUST properties, we'll also check that all items in the body/target array satisfy requirements for one of the body / target classes. We already have a schema that pretty much does this.
 
Do you envision people using such complex annotations as examples when running these tests?  I mean, I get that an implementation might be able to create such an annotation, but that is not something that is required by the spec.  The spec just says that simple annotations like like X and complex ones can be composed.  
 
 
Anyway, does this make sense?  I'm a little leery of uploading more schemas if I've got the wrong end of the stick. I'm also concerned that it will be easier to 'call' if the check for Annotation MUSTs is a schema (assertion file) rather than a test file – but maybe not for reporting purposes?
 
See above.  I think that your approach is fine.  I will redo the examples I put up there against the schema you uploaded so that you can see how I expect it to be structured.  Note that in order to do this I might need to change your schema so if something is not present it is NOT a failure.  Or just override those properties in the AssertionObject in the .test file. I will see what works best.
 
 
Basically the difference from a week ago, is that we're not doing separate checks for the presence of a key and then for its correctness using 2 different schemas (Rob, is this acceptable), we're reporting on each key individually, and the only complete failure is if an Annotation fails MUST requirements, or if the feature being tested for is not detectable (either because it wasn't implemented or because it was implemented in a manner that MUSTs for that feature were not implemented.
 
I agree with this approach.