Re: [Testing] Alternative approach to the test manifest structure

Thanks Shane, I didn't find the OR syntax for the test (as opposed to
within the schema itself).

Could you (or someone) give an example of how the structured tests might
work, as I must have missed it in the docs and current set of tests?  In
English, what I want to do is:

* Test whether there's a body property or not
  * If there is, test if it's a JSON string.
    * If it is, test that it's a URI
  * If there is, test if it's a JSON object.
    * If it is, test if it's a TextualBody
      * If it is, test whether there's a value property or not
        * If there is, test that it's a string
      * ...
    * If it is, test if it's a SpecificResource
      * ...
* Test whether there's a bodyValue property or not
  * If there is, test if it's a JSON string
* Otherwise raise a warning that there's no body

Where each of those is a separate schema, so they can be reused (e.g. value
is used on many sorts of resources)

Many thanks!

Rob


On Tue, Jul 12, 2016 at 4:13 PM, Shane McCarron <shane@spec-ops.io> wrote:

> Umm... I am not clear what problem you are trying to solve.  Regardless,
> you can do what you express with the current syntax (which is not a
> manifest).  Each .test file expresses one or more assertions that will be
> evaluated.  You can have "or" clauses in the declarative syntax so you can
> this OR this OR this OR this need to be true in order to satisfy the
> requirements of the test.
>
> On Tue, Jul 12, 2016 at 6:01 PM, Robert Sanderson <azaroth42@gmail.com>
> wrote:
>
>>
>> All,
>>
>> After playing around with the schemas over the weekend, with a view to
>> integrating them into my server implementation to validate the incoming
>> annotations, I ran into some issues:
>>
>> * The tests are designed for humans to read the error message, not for
>> machines to process the results ... some tests are okay to fail validation,
>> some aren't.
>>
>> * There doesn't seem to be a way to descend into the referenced resources
>> automatically.  You need to run the specific resource tests against the
>> specific resource by hand.
>>
>> * The processing patterns of the single with break or skip seems like it
>> could be extended ...
>>
>>
>> So what do people think about the following, if it's not too late to
>> change things:
>>
>> * Continue with atomic tests for presence of a property, and then a
>> separate one for the value of it
>> * But do that in the framework testing "manifest" by testing for
>> failure/success of the validation.
>>
>> For example, an automated system could descend into a SpecificResource as
>> the body by:
>>
>> * Test that body exists
>>   -- OnFail:  Warn (body is a SHOULD)
>>   -- OnSuccess:
>>       * Determine type of body
>>           -- uri?  OnSuccess:  Test URI-ness & goto next set of tests
>>                       OnFail: SpecificResource?
>>                               OnSuccess:  Descend into SpecificResource
>> tests
>>                               OnFail: TextualBody?
>>
>> And so forth.
>> The success/fail would be by the same $ref approach as the schema
>> includes, and offset from the schema itself, so it can be reused in
>> different parts of the overall set of tests.
>>
>> This would let us compose features at whatever level we think is
>> appropriate for reporting, and give a good validation suite for the model
>> that can be used completely programmatically by implementing the
>> success/fail runner.
>> [I did this runner already in python, it's a pretty easy piece of code,
>> as one might expect]
>>
>> Thoughts?
>>
>> Rob
>>
>> --
>> Rob Sanderson
>> Semantic Architect
>> The Getty Trust
>> Los Angeles, CA 90049
>>
>
>
>
> --
> Shane McCarron
> Projects Manager, Spec-Ops
>



-- 
Rob Sanderson
Semantic Architect
The Getty Trust
Los Angeles, CA 90049