Re: [Testing] Alternative approach to the test manifest structure

I did a little more on this example to illustrate the point more fully.
First look at the .test file:

{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
  "name": "There must be a body or a bodyValue property",
  "description": "Supply an example annotation that conforms to the basic
structure.",
  "ref": "https://www.w3.org/TR/annotation-model/#model",
  "assertions": [
    { "title": "There must be a body or a bodyValue property",
      "assertionType": "must",
      "expectedResult": "valid",
      "errorMessage": "Error: Annotation has no target relationships - at
least 1 is required. (Section 3.1)",
      "comparison": "or",
      "assertions": [
        "hasBody.json",
        "hasBodyValue.json"
        ]
    }
    ]
}


Then consider the two external .json files.  They don't need to be external
- it is just a little easier to read:

hasBody.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Has a body property",
  "description": "'http://www.w3.org/ns/anno.jsonld must be' an @context
value (Section 3.1)",
  "assertionType": "must",
  "expectedResult": "valid",
  "errorMessage": "There is no body property",
  "type": "object",
  "properties": {
    "body": {
      "oneOf": [
        { "$ref": "stringURI.json#" },
        { "type": "object",
          "oneOf": [
            { "$ref": "textualBody.json#" },
            { "$ref": "specificResource.json#" }
          ]
        }
      ]
    },
    "required": [ "body" ]
  }
}


hasBodyValue.json


{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Has a bodyValue property",
  "description": "'http://www.w3.org/ns/anno.jsonld must be' an @context
value (Section 3.1)",
  "assertionType": "must",
  "expectedResult": "valid",
  "errorMessage": "There is no bodyValue property",
  "type": "object",
  "properties": {
    "bodyValue": {
      "type": "string"
    },
    "required": [ "bodyValue" ]
  }
}


I haven't actually tested this, but I hope you can see the structure.  You
have a top level "or" to distinguish between the two main branches... then
for each branch I have a json file that uses the definitions you helpfully
provided and I select oneOf the options.


On Wed, Jul 13, 2016 at 1:42 PM, Shane McCarron <shane@spec-ops.io> wrote:

> The .test file syntax is defined at [1] (note that we are now living in
> the formal web-platform-test repository; yay!). This syntax indicates at
> the top level there is an "assertions" list.  This list can, of course,
> have only a single entry - where that entry is another list, a URI,
> Assertion Object, or a Condition Object (Condition Object is a
> specialization of an Assertion Object). An example of an "or" test is
> contained in the document referenced at [1] in the section on Condition
> Objects where it says:
>
>   "assertions": [
>     { "$schema": "http://json-schema.org/draft-04/schema#",
>       "title": "must have context or id",
>       "description": "A more complex example that allows one of many
> options to pass",
>       "assertions": [
>       { "title": "Condition Object",
>         "description": "A pseudo-test that will get a result from the
> aggregate of its children",
>         "assertionType": "must",
>         "expectedResult": "valid",
>         "errorMessage": "Error: None of the various options were present",
>         "compareWith": "or",
>         "assertions": [
>           "common/has_context.json",
>           "common/has_id.json"
>         ]
>       }
>       ]
>     }
>     ]
>
> In this example, there is a single "assertion" that has an "or" list of
> assertions within it.  Such a list permits the "or-ing" or the results of
> the embedded assertions - in this case that there is a context or there is
> an id.
>
> So, to get back to your request, you could that embedded "assertions" list
> as :
>
> "assertionType": "must",
> "errorMessage": "The annotation had neither a well formed body nor a well
> formed bodyValue property",
> "compareWith": "or",
> "assertions": [
>   { "title": "Has Body",
>     "description": "The annotation has a well formatted body property",
>     "assertions": [
>       "hasBody.json"
>     ]
>   },
>   { "title": "Has BodyValue",
>     "description": "The annotation has a well formatted bodyValue
> property",
>     "assertions": [
>       "hasBodyValue.json"
>     ]
>   }
> ]
>
>
> So it would evaluate those two things in an OR context and, if neither
> passed, report the errorMessage.
>
> The external files hasBody,json and hasBodyValue.json would contain the
> other logic:
>
> hasBody.json:
>
> { "$schema": "http://json-schema.org/draft-04/schema#",
>   "title": "is the Body property well formed",
>   "description": "Does the body property contain either a URI, a JSON
> Object that is a textual body, or a JSON Object that is a specific
> property",
>    "compareWith": "or",
>    "assertions": [
>      ...
>     ]
> }
>
> You get the idea.  I can probably create a working example.  However, this
> is a pretty complicated case.  Maybe we could start with something a little
> simpler to demonstrate that it does what you want and then work our way up
> to this?
>
> [1]
> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md
>
> On Wed, Jul 13, 2016 at 10:21 AM, Robert Sanderson <azaroth42@gmail.com>
> wrote:
>
>> Thanks Shane, I didn't find the OR syntax for the test (as opposed to
>> within the schema itself).
>>
>> Could you (or someone) give an example of how the structured tests might
>> work, as I must have missed it in the docs and current set of tests?  In
>> English, what I want to do is:
>>
>> * Test whether there's a body property or not
>>   * If there is, test if it's a JSON string.
>>     * If it is, test that it's a URI
>>   * If there is, test if it's a JSON object.
>>     * If it is, test if it's a TextualBody
>>       * If it is, test whether there's a value property or not
>>         * If there is, test that it's a string
>>       * ...
>>     * If it is, test if it's a SpecificResource
>>       * ...
>> * Test whether there's a bodyValue property or not
>>   * If there is, test if it's a JSON string
>> * Otherwise raise a warning that there's no body
>>
>> Where each of those is a separate schema, so they can be reused (e.g.
>> value is used on many sorts of resources)
>>
>> Many thanks!
>>
>> Rob
>>
>>
>> On Tue, Jul 12, 2016 at 4:13 PM, Shane McCarron <shane@spec-ops.io>
>> wrote:
>>
>>> Umm... I am not clear what problem you are trying to solve.  Regardless,
>>> you can do what you express with the current syntax (which is not a
>>> manifest).  Each .test file expresses one or more assertions that will be
>>> evaluated.  You can have "or" clauses in the declarative syntax so you can
>>> this OR this OR this OR this need to be true in order to satisfy the
>>> requirements of the test.
>>>
>>> On Tue, Jul 12, 2016 at 6:01 PM, Robert Sanderson <azaroth42@gmail.com>
>>> wrote:
>>>
>>>>
>>>> All,
>>>>
>>>> After playing around with the schemas over the weekend, with a view to
>>>> integrating them into my server implementation to validate the incoming
>>>> annotations, I ran into some issues:
>>>>
>>>> * The tests are designed for humans to read the error message, not for
>>>> machines to process the results ... some tests are okay to fail validation,
>>>> some aren't.
>>>>
>>>> * There doesn't seem to be a way to descend into the referenced
>>>> resources automatically.  You need to run the specific resource tests
>>>> against the specific resource by hand.
>>>>
>>>> * The processing patterns of the single with break or skip seems like
>>>> it could be extended ...
>>>>
>>>>
>>>> So what do people think about the following, if it's not too late to
>>>> change things:
>>>>
>>>> * Continue with atomic tests for presence of a property, and then a
>>>> separate one for the value of it
>>>> * But do that in the framework testing "manifest" by testing for
>>>> failure/success of the validation.
>>>>
>>>> For example, an automated system could descend into a SpecificResource
>>>> as the body by:
>>>>
>>>> * Test that body exists
>>>>   -- OnFail:  Warn (body is a SHOULD)
>>>>   -- OnSuccess:
>>>>       * Determine type of body
>>>>           -- uri?  OnSuccess:  Test URI-ness & goto next set of tests
>>>>                       OnFail: SpecificResource?
>>>>                               OnSuccess:  Descend into SpecificResource
>>>> tests
>>>>                               OnFail: TextualBody?
>>>>
>>>> And so forth.
>>>> The success/fail would be by the same $ref approach as the schema
>>>> includes, and offset from the schema itself, so it can be reused in
>>>> different parts of the overall set of tests.
>>>>
>>>> This would let us compose features at whatever level we think is
>>>> appropriate for reporting, and give a good validation suite for the model
>>>> that can be used completely programmatically by implementing the
>>>> success/fail runner.
>>>> [I did this runner already in python, it's a pretty easy piece of code,
>>>> as one might expect]
>>>>
>>>> Thoughts?
>>>>
>>>> Rob
>>>>
>>>> --
>>>> Rob Sanderson
>>>> Semantic Architect
>>>> The Getty Trust
>>>> Los Angeles, CA 90049
>>>>
>>>
>>>
>>>
>>> --
>>> Shane McCarron
>>> Projects Manager, Spec-Ops
>>>
>>
>>
>>
>> --
>> Rob Sanderson
>> Semantic Architect
>> The Getty Trust
>> Los Angeles, CA 90049
>>
>
>
>
> --
> Shane McCarron
> Projects Manager, Spec-Ops
>



-- 
Shane McCarron
Projects Manager, Spec-Ops