- From: Shane McCarron <shane@spec-ops.io>
- Date: Thu, 14 Jul 2016 18:27:49 -0500
- To: Robert Sanderson <azaroth42@gmail.com>
- Cc: Web Annotation <public-annotation@w3.org>
- Message-ID: <CAJdbnOAqo6+H6O0Y9KOZ1=dfvzpnLMUA_ahKw6+midNkZcrcdQ@mail.gmail.com>
I got the fix for this done this morning and checked in. Benjamin reviewed
it and it has been merged into WPT. It is deployed on testdev.spec-ops.io.
I have a more complex example that I am going to push into that tree so we
can talk about it in the meeting tomorrow.
On Thu, Jul 14, 2016 at 7:59 AM, Shane McCarron <shane@spec-ops.io> wrote:
> I filed a bug report against the JSONtest.js framework in WPT [1]. I am
> working to get it resolved this AM. Unfortunately I have a hard stop in
> about 3 hours. I hope to get it sorted out before then.
>
> [1] https://github.com/w3c/web-platform-tests/issues/3297
>
> On Wed, Jul 13, 2016 at 4:02 PM, Shane McCarron <shane@spec-ops.io> wrote:
>
>> It might not need to be that complex. I am working on making it work
>> right now... I found a little bug in the recursive processing of nested
>> assertion lists that I am trying to sort out.
>>
>> On Wed, Jul 13, 2016 at 3:55 PM, Robert Sanderson <azaroth42@gmail.com>
>> wrote:
>>
>>> Thanks for the examples Shane, they're very clear as to what they test :)
>>>
>>> So we would use ConditionObjects to handle the nesting, if I'm reading
>>> it correctly.
>>>
>>> I think the nesting of the test for body is something like:
>>>
>>> (body is present AND bodyValue is not present
>>> AND (body is a uri OR body is a TextualBody OR body is a
>>> SpecificResource OR body is a Resource)
>>> )
>>> OR (bodyValue is present AND body is not present AND bodyValue is a
>>> string)
>>> OR neither are present
>>>
>>> So, per
>>> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md#condition-objects,
>>> it would be something like:
>>>
>>> {
>>> "title": "Body"
>>> "compareWith": "or",
>>> "assertions": [
>>> {
>>> "title": "body is present and valid"
>>> "compareWith": "and",
>>> "assertions": [
>>> {
>>> "title": "body is present"
>>> "assertions": "/annotation/bodyPresent.json"
>>> },
>>> {
>>> "title": "bodyValue is not present",
>>> "assertions": "/annotation/bodyValueNotPresent.json"
>>> },
>>> {
>>> "title": "body value is correct",
>>> "compareWith": "or",
>>> "assertions": [
>>> {
>>> "title": "Body is a URI",
>>> "assertions": "/annotation/bodyUri.json"
>>> },
>>> {
>>> "title": "Body is a TextualBody",
>>> "assertions": "/annotation/bodyTextualBody.json",
>>> }
>>> ...
>>>
>>> etc.
>>>
>>> Am I on the right track? :)
>>>
>>> Thanks again,
>>>
>>> Rob
>>>
>>>
>>>
>>> On Wed, Jul 13, 2016 at 12:52 PM, Shane McCarron <shane@spec-ops.io>
>>> wrote:
>>>
>>>> I did a little more on this example to illustrate the point more
>>>> fully. First look at the .test file:
>>>>
>>>> {
>>>> "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
>>>> "name": "There must be a body or a bodyValue property",
>>>> "description": "Supply an example annotation that conforms to the
>>>> basic structure.",
>>>> "ref": "https://www.w3.org/TR/annotation-model/#model",
>>>> "assertions": [
>>>> { "title": "There must be a body or a bodyValue property",
>>>> "assertionType": "must",
>>>> "expectedResult": "valid",
>>>> "errorMessage": "Error: Annotation has no target relationships -
>>>> at least 1 is required. (Section 3.1)",
>>>> "comparison": "or",
>>>> "assertions": [
>>>> "hasBody.json",
>>>> "hasBodyValue.json"
>>>> ]
>>>> }
>>>> ]
>>>> }
>>>>
>>>>
>>>> Then consider the two external .json files. They don't need to be
>>>> external - it is just a little easier to read:
>>>>
>>>> hasBody.json
>>>>
>>>> {
>>>> "$schema": "http://json-schema.org/draft-04/schema#",
>>>> "title": "Has a body property",
>>>> "description": "'http://www.w3.org/ns/anno.jsonld must be' an
>>>> @context value (Section 3.1)",
>>>> "assertionType": "must",
>>>> "expectedResult": "valid",
>>>> "errorMessage": "There is no body property",
>>>> "type": "object",
>>>> "properties": {
>>>> "body": {
>>>> "oneOf": [
>>>> { "$ref": "stringURI.json#" },
>>>> { "type": "object",
>>>> "oneOf": [
>>>> { "$ref": "textualBody.json#" },
>>>> { "$ref": "specificResource.json#" }
>>>> ]
>>>> }
>>>> ]
>>>> },
>>>> "required": [ "body" ]
>>>> }
>>>> }
>>>>
>>>>
>>>> hasBodyValue.json
>>>>
>>>>
>>>> {
>>>> "$schema": "http://json-schema.org/draft-04/schema#",
>>>> "title": "Has a bodyValue property",
>>>> "description": "'http://www.w3.org/ns/anno.jsonld must be' an
>>>> @context value (Section 3.1)",
>>>> "assertionType": "must",
>>>> "expectedResult": "valid",
>>>> "errorMessage": "There is no bodyValue property",
>>>> "type": "object",
>>>> "properties": {
>>>> "bodyValue": {
>>>> "type": "string"
>>>> },
>>>> "required": [ "bodyValue" ]
>>>> }
>>>> }
>>>>
>>>>
>>>> I haven't actually tested this, but I hope you can see the structure.
>>>> You have a top level "or" to distinguish between the two main branches...
>>>> then for each branch I have a json file that uses the definitions you
>>>> helpfully provided and I select oneOf the options.
>>>>
>>>>
>>>> On Wed, Jul 13, 2016 at 1:42 PM, Shane McCarron <shane@spec-ops.io>
>>>> wrote:
>>>>
>>>>> The .test file syntax is defined at [1] (note that we are now living
>>>>> in the formal web-platform-test repository; yay!). This syntax indicates at
>>>>> the top level there is an "assertions" list. This list can, of course,
>>>>> have only a single entry - where that entry is another list, a URI,
>>>>> Assertion Object, or a Condition Object (Condition Object is a
>>>>> specialization of an Assertion Object). An example of an "or" test is
>>>>> contained in the document referenced at [1] in the section on Condition
>>>>> Objects where it says:
>>>>>
>>>>> "assertions": [
>>>>> { "$schema": "http://json-schema.org/draft-04/schema#",
>>>>> "title": "must have context or id",
>>>>> "description": "A more complex example that allows one of many
>>>>> options to pass",
>>>>> "assertions": [
>>>>> { "title": "Condition Object",
>>>>> "description": "A pseudo-test that will get a result from the
>>>>> aggregate of its children",
>>>>> "assertionType": "must",
>>>>> "expectedResult": "valid",
>>>>> "errorMessage": "Error: None of the various options were
>>>>> present",
>>>>> "compareWith": "or",
>>>>> "assertions": [
>>>>> "common/has_context.json",
>>>>> "common/has_id.json"
>>>>> ]
>>>>> }
>>>>> ]
>>>>> }
>>>>> ]
>>>>>
>>>>> In this example, there is a single "assertion" that has an "or" list
>>>>> of assertions within it. Such a list permits the "or-ing" or the results
>>>>> of the embedded assertions - in this case that there is a context or there
>>>>> is an id.
>>>>>
>>>>> So, to get back to your request, you could that embedded "assertions"
>>>>> list as :
>>>>>
>>>>> "assertionType": "must",
>>>>> "errorMessage": "The annotation had neither a well formed body nor a
>>>>> well formed bodyValue property",
>>>>> "compareWith": "or",
>>>>> "assertions": [
>>>>> { "title": "Has Body",
>>>>> "description": "The annotation has a well formatted body property",
>>>>> "assertions": [
>>>>> "hasBody.json"
>>>>> ]
>>>>> },
>>>>> { "title": "Has BodyValue",
>>>>> "description": "The annotation has a well formatted bodyValue
>>>>> property",
>>>>> "assertions": [
>>>>> "hasBodyValue.json"
>>>>> ]
>>>>> }
>>>>> ]
>>>>>
>>>>>
>>>>> So it would evaluate those two things in an OR context and, if neither
>>>>> passed, report the errorMessage.
>>>>>
>>>>> The external files hasBody,json and hasBodyValue.json would contain
>>>>> the other logic:
>>>>>
>>>>> hasBody.json:
>>>>>
>>>>> { "$schema": "http://json-schema.org/draft-04/schema#",
>>>>> "title": "is the Body property well formed",
>>>>> "description": "Does the body property contain either a URI, a JSON
>>>>> Object that is a textual body, or a JSON Object that is a specific
>>>>> property",
>>>>> "compareWith": "or",
>>>>> "assertions": [
>>>>> ...
>>>>> ]
>>>>> }
>>>>>
>>>>> You get the idea. I can probably create a working example. However,
>>>>> this is a pretty complicated case. Maybe we could start with something a
>>>>> little simpler to demonstrate that it does what you want and then work our
>>>>> way up to this?
>>>>>
>>>>> [1]
>>>>> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md
>>>>>
>>>>> On Wed, Jul 13, 2016 at 10:21 AM, Robert Sanderson <
>>>>> azaroth42@gmail.com> wrote:
>>>>>
>>>>>> Thanks Shane, I didn't find the OR syntax for the test (as opposed to
>>>>>> within the schema itself).
>>>>>>
>>>>>> Could you (or someone) give an example of how the structured tests
>>>>>> might work, as I must have missed it in the docs and current set of tests?
>>>>>> In English, what I want to do is:
>>>>>>
>>>>>> * Test whether there's a body property or not
>>>>>> * If there is, test if it's a JSON string.
>>>>>> * If it is, test that it's a URI
>>>>>> * If there is, test if it's a JSON object.
>>>>>> * If it is, test if it's a TextualBody
>>>>>> * If it is, test whether there's a value property or not
>>>>>> * If there is, test that it's a string
>>>>>> * ...
>>>>>> * If it is, test if it's a SpecificResource
>>>>>> * ...
>>>>>> * Test whether there's a bodyValue property or not
>>>>>> * If there is, test if it's a JSON string
>>>>>> * Otherwise raise a warning that there's no body
>>>>>>
>>>>>> Where each of those is a separate schema, so they can be reused (e.g.
>>>>>> value is used on many sorts of resources)
>>>>>>
>>>>>> Many thanks!
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 12, 2016 at 4:13 PM, Shane McCarron <shane@spec-ops.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Umm... I am not clear what problem you are trying to solve.
>>>>>>> Regardless, you can do what you express with the current syntax (which is
>>>>>>> not a manifest). Each .test file expresses one or more assertions that
>>>>>>> will be evaluated. You can have "or" clauses in the declarative syntax so
>>>>>>> you can this OR this OR this OR this need to be true in order to satisfy
>>>>>>> the requirements of the test.
>>>>>>>
>>>>>>> On Tue, Jul 12, 2016 at 6:01 PM, Robert Sanderson <
>>>>>>> azaroth42@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> All,
>>>>>>>>
>>>>>>>> After playing around with the schemas over the weekend, with a view
>>>>>>>> to integrating them into my server implementation to validate the incoming
>>>>>>>> annotations, I ran into some issues:
>>>>>>>>
>>>>>>>> * The tests are designed for humans to read the error message, not
>>>>>>>> for machines to process the results ... some tests are okay to fail
>>>>>>>> validation, some aren't.
>>>>>>>>
>>>>>>>> * There doesn't seem to be a way to descend into the referenced
>>>>>>>> resources automatically. You need to run the specific resource tests
>>>>>>>> against the specific resource by hand.
>>>>>>>>
>>>>>>>> * The processing patterns of the single with break or skip seems
>>>>>>>> like it could be extended ...
>>>>>>>>
>>>>>>>>
>>>>>>>> So what do people think about the following, if it's not too late
>>>>>>>> to change things:
>>>>>>>>
>>>>>>>> * Continue with atomic tests for presence of a property, and then a
>>>>>>>> separate one for the value of it
>>>>>>>> * But do that in the framework testing "manifest" by testing for
>>>>>>>> failure/success of the validation.
>>>>>>>>
>>>>>>>> For example, an automated system could descend into a
>>>>>>>> SpecificResource as the body by:
>>>>>>>>
>>>>>>>> * Test that body exists
>>>>>>>> -- OnFail: Warn (body is a SHOULD)
>>>>>>>> -- OnSuccess:
>>>>>>>> * Determine type of body
>>>>>>>> -- uri? OnSuccess: Test URI-ness & goto next set of
>>>>>>>> tests
>>>>>>>> OnFail: SpecificResource?
>>>>>>>> OnSuccess: Descend into
>>>>>>>> SpecificResource tests
>>>>>>>> OnFail: TextualBody?
>>>>>>>>
>>>>>>>> And so forth.
>>>>>>>> The success/fail would be by the same $ref approach as the schema
>>>>>>>> includes, and offset from the schema itself, so it can be reused in
>>>>>>>> different parts of the overall set of tests.
>>>>>>>>
>>>>>>>> This would let us compose features at whatever level we think is
>>>>>>>> appropriate for reporting, and give a good validation suite for the model
>>>>>>>> that can be used completely programmatically by implementing the
>>>>>>>> success/fail runner.
>>>>>>>> [I did this runner already in python, it's a pretty easy piece of
>>>>>>>> code, as one might expect]
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> Rob
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rob Sanderson
>>>>>>>> Semantic Architect
>>>>>>>> The Getty Trust
>>>>>>>> Los Angeles, CA 90049
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Shane McCarron
>>>>>>> Projects Manager, Spec-Ops
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rob Sanderson
>>>>>> Semantic Architect
>>>>>> The Getty Trust
>>>>>> Los Angeles, CA 90049
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane McCarron
>>>>> Projects Manager, Spec-Ops
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Shane McCarron
>>>> Projects Manager, Spec-Ops
>>>>
>>>
>>>
>>>
>>> --
>>> Rob Sanderson
>>> Semantic Architect
>>> The Getty Trust
>>> Los Angeles, CA 90049
>>>
>>
>>
>>
>> --
>> Shane McCarron
>> Projects Manager, Spec-Ops
>>
>
>
>
> --
> Shane McCarron
> Projects Manager, Spec-Ops
>
--
Shane McCarron
Projects Manager, Spec-Ops
Received on Thursday, 14 July 2016 23:28:46 UTC