Re: annotation data model -- running test scripts

Comments inline:

On Thu, Aug 11, 2016 at 11:01 AM, Timothy Cole <t-cole3@illinois.edu> wrote:

> Shane-
>
>
>
> We've made progress on rounding out the json schemas / assertions needed
> for testing the model.  Still more to generate, but now trying to create
> and generate the test scripts to understand better how that part of the
> process works.
>
>
>
> A.
>
> One glitch is that the test harness processes assertions (json schemas)
> as an entire file, i.e., does not seem to allow reference to individual
> schemas within a single .json file.  Can work around this, but wanted to
> make sure I wasn't missing something obvious.  So for example,
>
>
>
> 3.1-annotationMustKeys.json contains 6 sub-schemas in #/definitions/.  So
> I wrote the assertions part of the test script thusly:
>
>
>
>   "assertions": [
>
>     "annotations/3.1-annotationMustKeys.json#/definitions/
> annotationContextValidated",
>
>     "annotations/3.1-annotationMustKeys.json#/definitions/
> annotationIdValidated",
>
>     "annotations/3.1-annotationMustKeys.json#/definitions/
> annotationTypeValidated",
>
>     "annotations/3.1-annotationMustKeys.json#/
> definitions/notBodyBodyValue",
>
>     "annotations/3.1-annotationMustKeys.json#/definitions/targetKeyFound",
>
>     "annotations/3.1-annotationMustKeys.json#/definitions/targetObjectsRecognized"
>
>
>   ]
>
>
>
>                 The test harness ignored the fragment part of the ids.  If
> there's a way when specifying assertions to reference sub-schemas directly,
> please let me know. Again not critical, but would be nice for a couple of
> logistical reasons.
>

Assume this doesn't work.  I don't have the cycles right now to try to sort
it out and we are weeks behind.  Either put them in the definitions
directory and then reference them inline in the .test file, or split them
apart.


>
>
> B.
>
> Otherwise the tests worked as expected, catching the errors I introduced.
> When errors encountered AJV error messages appear in the JSON output from
> the test harness.  However, the schemas themselves include errorMessage
> values.  These values don't appear in the JSON output as best I can tell.
> Do these error messages appear elsewhere, or is there something different
> I should be doing to make these messages appear when a schema fails?
>

The error message should be attached to the AJV output.  I will look into
this.


>
>
> C.
>
> Could you clarify the meaning of the testType parameter?
>

It has no meaning.


>
>
> FYI, I will eliminate the old test scripts that reference schemas no
> longer present so as to streamline running of the tests. I expect to add
> several more manual tests in appropriate folders today.
>

Excellent.


>
>
> Thanks,
>
>
>
> Tim Cole
>
>
>
> .
>
>
>
>
>
> http://testdev.spec-ops.io:8000/annotation-model/
> annotations/3.1-model-musts-v2-manual.html
>



-- 
Shane McCarron
Projects Manager, Spec-Ops