Re: placeholder test files

Thanks for your detailed email.  I have put detailed comments inline.  Let
me preface everything by saying that this is almost all philosophy of
testing stuff.  There are indeed may ways to skin this cat.  None of them
are really wrong. And I agree that we should do everything in a consistent
manner to the extent that makes sense.  Having said that...

On Wed, Jul 20, 2016 at 4:37 PM, Timothy Cole <t-cole3@illinois.edu> wrote:

> Shane-
>
>
>
> I'm confused again, especially about the distinction between assertion
> files / schemas (*.json) and tests (*.test) and how we want to use them –
> realizing that there are multiple ways to skin a cat, but wanting to make
> sure we settle collectively on the same way. We're probably in the same
> ball park, but your examples make me think I heading in the opposite
> direction from you.
>
>
>
> So, Jacob, Janina, and I were going forward on the assumption that we
> needed individual assertion files (json schemas) for each of the
> properties defined in the Web Annotation model. Tests would reference these
> assertion files rather than embed schemas directly, and we would use the "
> onUnexpectedResult" : "failAndSkip", "onUnexpectedResult" : "
> failAndContinue", "onUnexpectedResult" : "succeedAndSkip", "
> onUnexpectedResult" : "suceedAndContinue" to manage the flow of
> assertions checked and would report error messages, etc.   The advantage
> of not embedding the schemas in the tests was that same keys are used on
> multiple objects defined in the model and so reusability should be
> facilitated.
>

Sure.  That's a fine way to go.   I will get into some detail below, but in
general I would say that external schema are preferred in any case where
there is possible reuse. I was putting them in line in my examples because
it makes the example easier to comprehend for some.

As to the flow of assertions and running tests... in general I believe it
is a good idea to evaluate all of the assertions in a test unless doing so
would result in catastrophe (e.g., testing laser firing controls and
discovering that the off control is disabled should result in an abort
because you would start firing the laser and burn down the building.  This
actually happened to me once.)  So in the case of our tests, when
evaluating any message, go ahead and test all of the bits that might be in
a message of a given shape.  If some piece is mandatory and missing or
wrong, fail the test.  If some piece is optional and missing, pass.  If
some piece is optional and present and wrong, fail.  This is particularly
important with regard to reporting.  In order for the reports to generate
cleanly and all of the data to line up well, all of the tests need to be
run by every platform.  I hope that is clear and I expect we agree fully.

More details below:



>
> Toward that end we uploaded json schemas yesterday designed to detect  and
> validate (to a limited extent) format, language, processingLanguage,
> textDirection, id. The first four of these (variously should and may) can
> appear on Textual Bodies, External Web Resource descriptions that appear as
> Bodies, Targets, SpecificResource Sources, items in Choice, List,
> Composite, and Independents. id can appear on these and on additional
> objects. We also uploaded a few schemas that can be used as assertions in a
> test to detect a Textual Body (using presence of the value key in certain
> conditions [a must] rather than the type value, since the type value for
> Textual Bodies is only a Should), as well as schemas designed to recognize
> the use of Choice, List, Composite or Independents individually (but are
> working on more schemas for these and for Specific Resources).  There
> were previously schemas uploaded to check for presence and correctness of
> @context, id, "type": "Annotation", target(s), etc., though I'm not sure
> they're in the right place. Our assumption was that other schema files and
> tests would then be written to leverage these individual assertion files
> (i.e., schemas), e.g.,
>

I think that is a great approach.  I was working in parallel with you and
embedded my assertions in some cases.  But there is no reason to do that.
I would rather reference external schema definitions - in particular so
that as those are refined the refinements are reflected everywhere.


>
> Given a purported Annotation, a test for Embedded Textual Body (if
> selected) would proceed something like:
>
>
>
> 1. Reference an assertion file (schema) that checked for all annotation
> MUSTS (referencing the @context, id, … assertion files and providing error
> messages that enumerate failings agains Annotation MUSTS as appropriate)
>
>      "onUnexpectedResult" : "failAndSkip" – json submitted does not meet
> MUST requirements to be an annotation, no further testing.
>

Disagree here.  There is no reason to stop.  All of the assertions are
embedded in or referenced from the same test case file (.test) so they can
be easily run and there is no reason to not run them as there is no cost.

2. Reference a schema to detect body(ies) of the Annotation that satisfy
> the MUST requirement for Embedded Textual body.
>
>      "onUnexpectedResult" : "failAndSkip" – json submitted does not
> implement Embedded Textual Body feature of the model, no further testing of
> Embedded Textual Body features.
>

I think I disagree here as well. Because as I said above, each platform
should run the same tests so we can do apples to apples comparisons.  Also
because there might be a small error in this part but the rest of the
annotation might be okay. Or, more importantly, there might be other errors
and if an implementer is fixing things based upon this report, they might
as well fix everything all at once.


>
> 3. Check if the Embedded Textual Body has a language property, if yes,
> then the Annotation's use of Textual Body implements the language feature
> for that kind of body.
>
>      "onUnexpectedResult" : "failAndContinue" – warn that language (a
> Should) was not implemented
>
>
>
> 4. Check if the Embedded Textual Body has a format property, if yes, then
> the Annotation's use of Textual Body implements the format feature.
>
>      "onUnexpectedResult" : "failAndContinue" – warn that format (a
> Should) was not implemented
>

This is not a conformance failure.  It might be valuable information for
the implementer.  But it is not a FAIL if the feature is not present.  It
is only a FAIL if the feature is present and implemented wrong.  Otherwise
it is a PASS.  I am checking with the WPT gurus about how they normally
handle SHOULD assertions in conformance tests.  We need to be consistent
with common use I think.


>
>
> 5. Check if the Embedded Textual Body has a processingLanguage property,
> if yes, then the Annotation's use of Textual Body implements the
> processingLanguage feature.
>
>      "onUnexpectedResult" : "failAndContinue" – inform that
> processingLanguage (a May) was not implemented
>
>
>
> 6. Check if the Embedded Textual Body has a textDirection property, if
> yes, then the Annotation's use of Textual Body implements the
> textDirection feature.
>
>      "onUnexpectedResult" : "failAndContinue" – inform that textDirection
> (a May) was not implemented
>
>
>
> 7. Check if the Embedded Textual Body has a id property, if yes, then the
> Annotation's use of Textual Body implements the id feature.
>
>      "onUnexpectedResult" : "failAndContinue" – inform that id (a May)
> was not implemented
>

A MAY was not implemented is not a FAIL.  Unfortunately, there does not
seem to be a clear way to send information along with a PASS result.  I am
going to dig into that.  If there is a way then we could extend our
declarative syntax to include "passAndMessageAndContinue" that adds a
message or something. Anyway, I will check into that for SHOULD and MAY
assertions - I appreciate what you are trying to do but we can't mark then
as FAIL if the test is indeed passing.  That would make our implementation
report look like... well, like everything was failing.


>
> Similar logic for checking for External Web Resources as bodies, but a
> separate test to see if EWR were implemented as a target. EWRs can also be
> SpecificResource sources. Textual Bodies and EWRs can be items in a
> Choice, List, Composite or Independents arrays.
>
>
>
> The test logic gets longer for Bodies or Targets that are
> SpecificResources, Choice, List, Composite or Independents, but the basic
> flow looks to be similar.
>

Sure - makes sense.


>
>
> When body or target is an array of mixed body/target types, this approach
> can only report on items matching the pattern being checked, i.e., only the
> Embedded Textual Bodies in an array of mixed Textual Bodies and External
> Web Resources – so we might run into an issue of reporting that annotation
> implemented Textual Bodies (i.e., correctly) but also had bodies that did
> not match any valid bodies. To avoid this, presumably, just as we will
> check all annotations for Annotation MUST properties, we'll also check that
> all items in the body/target array satisfy requirements for one of the body
> / target classes. We already have a schema that pretty much does this.
>

Do you envision people using such complex annotations as examples when
running these tests?  I mean, I get that an implementation might be able to
create such an annotation, but that is not something that is required by
the spec.  The spec just says that simple annotations like like X and
complex ones can be composed.


>
>
> Anyway, does this make sense?  I'm a little leery of uploading more
> schemas if I've got the wrong end of the stick. I'm also concerned that it
> will be easier to 'call' if the check for Annotation MUSTs is a schema
> (assertion file) rather than a test file – but maybe not for reporting
> purposes?
>

See above.  I think that your approach is fine.  I will redo the examples I
put up there against the schema you uploaded so that you can see how I
expect it to be structured.  Note that in order to do this I might need to
change your schema so if something is not present it is NOT a failure.  Or
just override those properties in the AssertionObject in the .test file. I
will see what works best.


>
>
> Basically the difference from a week ago, is that we're not doing separate
> checks for the presence of a key and then for its correctness using 2
> different schemas (Rob, is this acceptable), we're reporting on each key
> individually, and the only complete failure is if an Annotation fails MUST
> requirements, or if the feature being tested for is not detectable (either
> because it wasn't implemented or because it was implemented in a manner
> that MUSTs for that feature were not implemented.
>

I agree with this approach.