RE: annotation data model -- running test scripts

Okay, broke a selection of the individual schemas / assertions into individual files.  Helped in that I now see the name of each assertion one-by-one and with results, but I don't see the errorMessage, and while expectedResult seems to operate as expected, onUnexpectedResult seems to have no effect, regardless of whether the value is failAndContinue and succeedAndContinue, the test harness says fail if result is unexpected.

For the example tests so far, see:

http://testdev.spec-ops.io:8000/annotation-model/annotations/3.1-model-musts-v3-manual.html

http://testdev.spec-ops.io:8000/annotation-model/bodiesTargets/3.2-model-manual.html

or run both old (some of which reference schemas no longer available) and new tests by going to
http://testdev.spec-ops.io:8000/tools/runner/index.html

and specifying for path either /annotation-model/annotations or /annotation-model/bodiesTargets

We can discuss tomorrow...

-Tim Cole



________________________________
From: Shane McCarron [shane@spec-ops.io]
Sent: Thursday, August 11, 2016 15:41
To: Cole, Timothy W
Cc: W3C Public Annotation List
Subject: Re: annotation data model -- running test scripts

Comments inline:

On Thu, Aug 11, 2016 at 11:01 AM, Timothy Cole <t-cole3@illinois.edu<mailto:t-cole3@illinois.edu>> wrote:
Shane-

We've made progress on rounding out the json schemas / assertions needed for testing the model.  Still more to generate, but now trying to create and generate the test scripts to understand better how that part of the process works.

A.
One glitch is that the test harness processes assertions (json schemas) as an entire file, i.e., does not seem to allow reference to individual schemas within a single .json file.  Can work around this, but wanted to make sure I wasn't missing something obvious.  So for example,

3.1-annotationMustKeys.json contains 6 sub-schemas in #/definitions/.  So I wrote the assertions part of the test script thusly:

  "assertions": [
    "annotations/3.1-annotationMustKeys.json#/definitions/annotationContextValidated",
    "annotations/3.1-annotationMustKeys.json#/definitions/annotationIdValidated",
    "annotations/3.1-annotationMustKeys.json#/definitions/annotationTypeValidated",
    "annotations/3.1-annotationMustKeys.json#/definitions/notBodyBodyValue",
    "annotations/3.1-annotationMustKeys.json#/definitions/targetKeyFound",
    "annotations/3.1-annotationMustKeys.json#/definitions/targetObjectsRecognized"
  ]

                The test harness ignored the fragment part of the ids.  If there's a way when specifying assertions to reference sub-schemas directly, please let me know. Again not critical, but would be nice for a couple of logistical reasons.

Assume this doesn't work.  I don't have the cycles right now to try to sort it out and we are weeks behind.  Either put them in the definitions directory and then reference them inline in the .test file, or split them apart.


B.
Otherwise the tests worked as expected, catching the errors I introduced. When errors encountered AJV error messages appear in the JSON output from the test harness.  However, the schemas themselves include errorMessage values.  These values don't appear in the JSON output as best I can tell.  Do these error messages appear elsewhere, or is there something different I should be doing to make these messages appear when a schema fails?

The error message should be attached to the AJV output.  I will look into this.


C.
Could you clarify the meaning of the testType parameter?

It has no meaning.


FYI, I will eliminate the old test scripts that reference schemas no longer present so as to streamline running of the tests. I expect to add several more manual tests in appropriate folders today.

Excellent.


Thanks,

Tim Cole

.


http://testdev.spec-ops.io:8000/annotation-model/annotations/3.1-model-musts-v2-manual.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__testdev.spec-2Dops.io-3A8000_annotation-2Dmodel_annotations_3.1-2Dmodel-2Dmusts-2Dv2-2Dmanual.html&d=CwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=zjI0r-H6xRs5fYf2_jJkju6US9ijk0nLw4ns2nuwU2k&m=S8Cs42jrBNgpn-LfQzqWIMclze9RizEmBM1IyBcqj50&s=3MXxzrM2NiZ2PeUvujQvKTjZ6H0JBXFAmXVVwssANLk&e=>



--
Shane McCarron
Projects Manager, Spec-Ops