- From: Shane McCarron <shane@spec-ops.io>
- Date: Wed, 13 Jul 2016 16:02:36 -0500
- To: Robert Sanderson <azaroth42@gmail.com>
- Cc: Web Annotation <public-annotation@w3.org>
- Message-ID: <CAJdbnOD-3P=4_hk=SwTFPSLeFg0SwcGUCx+ookwOcqDkyEoo5A@mail.gmail.com>
It might not need to be that complex. I am working on making it work right
now... I found a little bug in the recursive processing of nested
assertion lists that I am trying to sort out.
On Wed, Jul 13, 2016 at 3:55 PM, Robert Sanderson <azaroth42@gmail.com>
wrote:
> Thanks for the examples Shane, they're very clear as to what they test :)
>
> So we would use ConditionObjects to handle the nesting, if I'm reading it
> correctly.
>
> I think the nesting of the test for body is something like:
>
> (body is present AND bodyValue is not present
> AND (body is a uri OR body is a TextualBody OR body is a
> SpecificResource OR body is a Resource)
> )
> OR (bodyValue is present AND body is not present AND bodyValue is a string)
> OR neither are present
>
> So, per
> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md#condition-objects,
> it would be something like:
>
> {
> "title": "Body"
> "compareWith": "or",
> "assertions": [
> {
> "title": "body is present and valid"
> "compareWith": "and",
> "assertions": [
> {
> "title": "body is present"
> "assertions": "/annotation/bodyPresent.json"
> },
> {
> "title": "bodyValue is not present",
> "assertions": "/annotation/bodyValueNotPresent.json"
> },
> {
> "title": "body value is correct",
> "compareWith": "or",
> "assertions": [
> {
> "title": "Body is a URI",
> "assertions": "/annotation/bodyUri.json"
> },
> {
> "title": "Body is a TextualBody",
> "assertions": "/annotation/bodyTextualBody.json",
> }
> ...
>
> etc.
>
> Am I on the right track? :)
>
> Thanks again,
>
> Rob
>
>
>
> On Wed, Jul 13, 2016 at 12:52 PM, Shane McCarron <shane@spec-ops.io>
> wrote:
>
>> I did a little more on this example to illustrate the point more fully.
>> First look at the .test file:
>>
>> {
>> "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
>> "name": "There must be a body or a bodyValue property",
>> "description": "Supply an example annotation that conforms to the basic
>> structure.",
>> "ref": "https://www.w3.org/TR/annotation-model/#model",
>> "assertions": [
>> { "title": "There must be a body or a bodyValue property",
>> "assertionType": "must",
>> "expectedResult": "valid",
>> "errorMessage": "Error: Annotation has no target relationships - at
>> least 1 is required. (Section 3.1)",
>> "comparison": "or",
>> "assertions": [
>> "hasBody.json",
>> "hasBodyValue.json"
>> ]
>> }
>> ]
>> }
>>
>>
>> Then consider the two external .json files. They don't need to be
>> external - it is just a little easier to read:
>>
>> hasBody.json
>>
>> {
>> "$schema": "http://json-schema.org/draft-04/schema#",
>> "title": "Has a body property",
>> "description": "'http://www.w3.org/ns/anno.jsonld must be' an @context
>> value (Section 3.1)",
>> "assertionType": "must",
>> "expectedResult": "valid",
>> "errorMessage": "There is no body property",
>> "type": "object",
>> "properties": {
>> "body": {
>> "oneOf": [
>> { "$ref": "stringURI.json#" },
>> { "type": "object",
>> "oneOf": [
>> { "$ref": "textualBody.json#" },
>> { "$ref": "specificResource.json#" }
>> ]
>> }
>> ]
>> },
>> "required": [ "body" ]
>> }
>> }
>>
>>
>> hasBodyValue.json
>>
>>
>> {
>> "$schema": "http://json-schema.org/draft-04/schema#",
>> "title": "Has a bodyValue property",
>> "description": "'http://www.w3.org/ns/anno.jsonld must be' an @context
>> value (Section 3.1)",
>> "assertionType": "must",
>> "expectedResult": "valid",
>> "errorMessage": "There is no bodyValue property",
>> "type": "object",
>> "properties": {
>> "bodyValue": {
>> "type": "string"
>> },
>> "required": [ "bodyValue" ]
>> }
>> }
>>
>>
>> I haven't actually tested this, but I hope you can see the structure.
>> You have a top level "or" to distinguish between the two main branches...
>> then for each branch I have a json file that uses the definitions you
>> helpfully provided and I select oneOf the options.
>>
>>
>> On Wed, Jul 13, 2016 at 1:42 PM, Shane McCarron <shane@spec-ops.io>
>> wrote:
>>
>>> The .test file syntax is defined at [1] (note that we are now living in
>>> the formal web-platform-test repository; yay!). This syntax indicates at
>>> the top level there is an "assertions" list. This list can, of course,
>>> have only a single entry - where that entry is another list, a URI,
>>> Assertion Object, or a Condition Object (Condition Object is a
>>> specialization of an Assertion Object). An example of an "or" test is
>>> contained in the document referenced at [1] in the section on Condition
>>> Objects where it says:
>>>
>>> "assertions": [
>>> { "$schema": "http://json-schema.org/draft-04/schema#",
>>> "title": "must have context or id",
>>> "description": "A more complex example that allows one of many
>>> options to pass",
>>> "assertions": [
>>> { "title": "Condition Object",
>>> "description": "A pseudo-test that will get a result from the
>>> aggregate of its children",
>>> "assertionType": "must",
>>> "expectedResult": "valid",
>>> "errorMessage": "Error: None of the various options were
>>> present",
>>> "compareWith": "or",
>>> "assertions": [
>>> "common/has_context.json",
>>> "common/has_id.json"
>>> ]
>>> }
>>> ]
>>> }
>>> ]
>>>
>>> In this example, there is a single "assertion" that has an "or" list of
>>> assertions within it. Such a list permits the "or-ing" or the results of
>>> the embedded assertions - in this case that there is a context or there is
>>> an id.
>>>
>>> So, to get back to your request, you could that embedded "assertions"
>>> list as :
>>>
>>> "assertionType": "must",
>>> "errorMessage": "The annotation had neither a well formed body nor a
>>> well formed bodyValue property",
>>> "compareWith": "or",
>>> "assertions": [
>>> { "title": "Has Body",
>>> "description": "The annotation has a well formatted body property",
>>> "assertions": [
>>> "hasBody.json"
>>> ]
>>> },
>>> { "title": "Has BodyValue",
>>> "description": "The annotation has a well formatted bodyValue
>>> property",
>>> "assertions": [
>>> "hasBodyValue.json"
>>> ]
>>> }
>>> ]
>>>
>>>
>>> So it would evaluate those two things in an OR context and, if neither
>>> passed, report the errorMessage.
>>>
>>> The external files hasBody,json and hasBodyValue.json would contain the
>>> other logic:
>>>
>>> hasBody.json:
>>>
>>> { "$schema": "http://json-schema.org/draft-04/schema#",
>>> "title": "is the Body property well formed",
>>> "description": "Does the body property contain either a URI, a JSON
>>> Object that is a textual body, or a JSON Object that is a specific
>>> property",
>>> "compareWith": "or",
>>> "assertions": [
>>> ...
>>> ]
>>> }
>>>
>>> You get the idea. I can probably create a working example. However,
>>> this is a pretty complicated case. Maybe we could start with something a
>>> little simpler to demonstrate that it does what you want and then work our
>>> way up to this?
>>>
>>> [1]
>>> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md
>>>
>>> On Wed, Jul 13, 2016 at 10:21 AM, Robert Sanderson <azaroth42@gmail.com>
>>> wrote:
>>>
>>>> Thanks Shane, I didn't find the OR syntax for the test (as opposed to
>>>> within the schema itself).
>>>>
>>>> Could you (or someone) give an example of how the structured tests
>>>> might work, as I must have missed it in the docs and current set of tests?
>>>> In English, what I want to do is:
>>>>
>>>> * Test whether there's a body property or not
>>>> * If there is, test if it's a JSON string.
>>>> * If it is, test that it's a URI
>>>> * If there is, test if it's a JSON object.
>>>> * If it is, test if it's a TextualBody
>>>> * If it is, test whether there's a value property or not
>>>> * If there is, test that it's a string
>>>> * ...
>>>> * If it is, test if it's a SpecificResource
>>>> * ...
>>>> * Test whether there's a bodyValue property or not
>>>> * If there is, test if it's a JSON string
>>>> * Otherwise raise a warning that there's no body
>>>>
>>>> Where each of those is a separate schema, so they can be reused (e.g.
>>>> value is used on many sorts of resources)
>>>>
>>>> Many thanks!
>>>>
>>>> Rob
>>>>
>>>>
>>>> On Tue, Jul 12, 2016 at 4:13 PM, Shane McCarron <shane@spec-ops.io>
>>>> wrote:
>>>>
>>>>> Umm... I am not clear what problem you are trying to solve.
>>>>> Regardless, you can do what you express with the current syntax (which is
>>>>> not a manifest). Each .test file expresses one or more assertions that
>>>>> will be evaluated. You can have "or" clauses in the declarative syntax so
>>>>> you can this OR this OR this OR this need to be true in order to satisfy
>>>>> the requirements of the test.
>>>>>
>>>>> On Tue, Jul 12, 2016 at 6:01 PM, Robert Sanderson <azaroth42@gmail.com
>>>>> > wrote:
>>>>>
>>>>>>
>>>>>> All,
>>>>>>
>>>>>> After playing around with the schemas over the weekend, with a view
>>>>>> to integrating them into my server implementation to validate the incoming
>>>>>> annotations, I ran into some issues:
>>>>>>
>>>>>> * The tests are designed for humans to read the error message, not
>>>>>> for machines to process the results ... some tests are okay to fail
>>>>>> validation, some aren't.
>>>>>>
>>>>>> * There doesn't seem to be a way to descend into the referenced
>>>>>> resources automatically. You need to run the specific resource tests
>>>>>> against the specific resource by hand.
>>>>>>
>>>>>> * The processing patterns of the single with break or skip seems like
>>>>>> it could be extended ...
>>>>>>
>>>>>>
>>>>>> So what do people think about the following, if it's not too late to
>>>>>> change things:
>>>>>>
>>>>>> * Continue with atomic tests for presence of a property, and then a
>>>>>> separate one for the value of it
>>>>>> * But do that in the framework testing "manifest" by testing for
>>>>>> failure/success of the validation.
>>>>>>
>>>>>> For example, an automated system could descend into a
>>>>>> SpecificResource as the body by:
>>>>>>
>>>>>> * Test that body exists
>>>>>> -- OnFail: Warn (body is a SHOULD)
>>>>>> -- OnSuccess:
>>>>>> * Determine type of body
>>>>>> -- uri? OnSuccess: Test URI-ness & goto next set of tests
>>>>>> OnFail: SpecificResource?
>>>>>> OnSuccess: Descend into
>>>>>> SpecificResource tests
>>>>>> OnFail: TextualBody?
>>>>>>
>>>>>> And so forth.
>>>>>> The success/fail would be by the same $ref approach as the schema
>>>>>> includes, and offset from the schema itself, so it can be reused in
>>>>>> different parts of the overall set of tests.
>>>>>>
>>>>>> This would let us compose features at whatever level we think is
>>>>>> appropriate for reporting, and give a good validation suite for the model
>>>>>> that can be used completely programmatically by implementing the
>>>>>> success/fail runner.
>>>>>> [I did this runner already in python, it's a pretty easy piece of
>>>>>> code, as one might expect]
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>> --
>>>>>> Rob Sanderson
>>>>>> Semantic Architect
>>>>>> The Getty Trust
>>>>>> Los Angeles, CA 90049
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane McCarron
>>>>> Projects Manager, Spec-Ops
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Rob Sanderson
>>>> Semantic Architect
>>>> The Getty Trust
>>>> Los Angeles, CA 90049
>>>>
>>>
>>>
>>>
>>> --
>>> Shane McCarron
>>> Projects Manager, Spec-Ops
>>>
>>
>>
>>
>> --
>> Shane McCarron
>> Projects Manager, Spec-Ops
>>
>
>
>
> --
> Rob Sanderson
> Semantic Architect
> The Getty Trust
> Los Angeles, CA 90049
>
--
Shane McCarron
Projects Manager, Spec-Ops
Received on Wednesday, 13 July 2016 21:03:33 UTC