- From: Shane McCarron <shane@spec-ops.io>
- Date: Thu, 14 Jul 2016 07:59:11 -0500
- To: Robert Sanderson <azaroth42@gmail.com>
- Cc: Web Annotation <public-annotation@w3.org>
- Message-ID: <CAJdbnOCCgiFWr2n0DOXT6DZp5krNhgQ3kqaMfYZy6HezaULNbg@mail.gmail.com>
I filed a bug report against the JSONtest.js framework in WPT [1]. I am
working to get it resolved this AM. Unfortunately I have a hard stop in
about 3 hours. I hope to get it sorted out before then.
[1] https://github.com/w3c/web-platform-tests/issues/3297
On Wed, Jul 13, 2016 at 4:02 PM, Shane McCarron <shane@spec-ops.io> wrote:
> It might not need to be that complex. I am working on making it work
> right now... I found a little bug in the recursive processing of nested
> assertion lists that I am trying to sort out.
>
> On Wed, Jul 13, 2016 at 3:55 PM, Robert Sanderson <azaroth42@gmail.com>
> wrote:
>
>> Thanks for the examples Shane, they're very clear as to what they test :)
>>
>> So we would use ConditionObjects to handle the nesting, if I'm reading it
>> correctly.
>>
>> I think the nesting of the test for body is something like:
>>
>> (body is present AND bodyValue is not present
>> AND (body is a uri OR body is a TextualBody OR body is a
>> SpecificResource OR body is a Resource)
>> )
>> OR (bodyValue is present AND body is not present AND bodyValue is a
>> string)
>> OR neither are present
>>
>> So, per
>> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md#condition-objects,
>> it would be something like:
>>
>> {
>> "title": "Body"
>> "compareWith": "or",
>> "assertions": [
>> {
>> "title": "body is present and valid"
>> "compareWith": "and",
>> "assertions": [
>> {
>> "title": "body is present"
>> "assertions": "/annotation/bodyPresent.json"
>> },
>> {
>> "title": "bodyValue is not present",
>> "assertions": "/annotation/bodyValueNotPresent.json"
>> },
>> {
>> "title": "body value is correct",
>> "compareWith": "or",
>> "assertions": [
>> {
>> "title": "Body is a URI",
>> "assertions": "/annotation/bodyUri.json"
>> },
>> {
>> "title": "Body is a TextualBody",
>> "assertions": "/annotation/bodyTextualBody.json",
>> }
>> ...
>>
>> etc.
>>
>> Am I on the right track? :)
>>
>> Thanks again,
>>
>> Rob
>>
>>
>>
>> On Wed, Jul 13, 2016 at 12:52 PM, Shane McCarron <shane@spec-ops.io>
>> wrote:
>>
>>> I did a little more on this example to illustrate the point more fully.
>>> First look at the .test file:
>>>
>>> {
>>> "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
>>> "name": "There must be a body or a bodyValue property",
>>> "description": "Supply an example annotation that conforms to the
>>> basic structure.",
>>> "ref": "https://www.w3.org/TR/annotation-model/#model",
>>> "assertions": [
>>> { "title": "There must be a body or a bodyValue property",
>>> "assertionType": "must",
>>> "expectedResult": "valid",
>>> "errorMessage": "Error: Annotation has no target relationships -
>>> at least 1 is required. (Section 3.1)",
>>> "comparison": "or",
>>> "assertions": [
>>> "hasBody.json",
>>> "hasBodyValue.json"
>>> ]
>>> }
>>> ]
>>> }
>>>
>>>
>>> Then consider the two external .json files. They don't need to be
>>> external - it is just a little easier to read:
>>>
>>> hasBody.json
>>>
>>> {
>>> "$schema": "http://json-schema.org/draft-04/schema#",
>>> "title": "Has a body property",
>>> "description": "'http://www.w3.org/ns/anno.jsonld must be' an
>>> @context value (Section 3.1)",
>>> "assertionType": "must",
>>> "expectedResult": "valid",
>>> "errorMessage": "There is no body property",
>>> "type": "object",
>>> "properties": {
>>> "body": {
>>> "oneOf": [
>>> { "$ref": "stringURI.json#" },
>>> { "type": "object",
>>> "oneOf": [
>>> { "$ref": "textualBody.json#" },
>>> { "$ref": "specificResource.json#" }
>>> ]
>>> }
>>> ]
>>> },
>>> "required": [ "body" ]
>>> }
>>> }
>>>
>>>
>>> hasBodyValue.json
>>>
>>>
>>> {
>>> "$schema": "http://json-schema.org/draft-04/schema#",
>>> "title": "Has a bodyValue property",
>>> "description": "'http://www.w3.org/ns/anno.jsonld must be' an
>>> @context value (Section 3.1)",
>>> "assertionType": "must",
>>> "expectedResult": "valid",
>>> "errorMessage": "There is no bodyValue property",
>>> "type": "object",
>>> "properties": {
>>> "bodyValue": {
>>> "type": "string"
>>> },
>>> "required": [ "bodyValue" ]
>>> }
>>> }
>>>
>>>
>>> I haven't actually tested this, but I hope you can see the structure.
>>> You have a top level "or" to distinguish between the two main branches...
>>> then for each branch I have a json file that uses the definitions you
>>> helpfully provided and I select oneOf the options.
>>>
>>>
>>> On Wed, Jul 13, 2016 at 1:42 PM, Shane McCarron <shane@spec-ops.io>
>>> wrote:
>>>
>>>> The .test file syntax is defined at [1] (note that we are now living in
>>>> the formal web-platform-test repository; yay!). This syntax indicates at
>>>> the top level there is an "assertions" list. This list can, of course,
>>>> have only a single entry - where that entry is another list, a URI,
>>>> Assertion Object, or a Condition Object (Condition Object is a
>>>> specialization of an Assertion Object). An example of an "or" test is
>>>> contained in the document referenced at [1] in the section on Condition
>>>> Objects where it says:
>>>>
>>>> "assertions": [
>>>> { "$schema": "http://json-schema.org/draft-04/schema#",
>>>> "title": "must have context or id",
>>>> "description": "A more complex example that allows one of many
>>>> options to pass",
>>>> "assertions": [
>>>> { "title": "Condition Object",
>>>> "description": "A pseudo-test that will get a result from the
>>>> aggregate of its children",
>>>> "assertionType": "must",
>>>> "expectedResult": "valid",
>>>> "errorMessage": "Error: None of the various options were
>>>> present",
>>>> "compareWith": "or",
>>>> "assertions": [
>>>> "common/has_context.json",
>>>> "common/has_id.json"
>>>> ]
>>>> }
>>>> ]
>>>> }
>>>> ]
>>>>
>>>> In this example, there is a single "assertion" that has an "or" list of
>>>> assertions within it. Such a list permits the "or-ing" or the results of
>>>> the embedded assertions - in this case that there is a context or there is
>>>> an id.
>>>>
>>>> So, to get back to your request, you could that embedded "assertions"
>>>> list as :
>>>>
>>>> "assertionType": "must",
>>>> "errorMessage": "The annotation had neither a well formed body nor a
>>>> well formed bodyValue property",
>>>> "compareWith": "or",
>>>> "assertions": [
>>>> { "title": "Has Body",
>>>> "description": "The annotation has a well formatted body property",
>>>> "assertions": [
>>>> "hasBody.json"
>>>> ]
>>>> },
>>>> { "title": "Has BodyValue",
>>>> "description": "The annotation has a well formatted bodyValue
>>>> property",
>>>> "assertions": [
>>>> "hasBodyValue.json"
>>>> ]
>>>> }
>>>> ]
>>>>
>>>>
>>>> So it would evaluate those two things in an OR context and, if neither
>>>> passed, report the errorMessage.
>>>>
>>>> The external files hasBody,json and hasBodyValue.json would contain the
>>>> other logic:
>>>>
>>>> hasBody.json:
>>>>
>>>> { "$schema": "http://json-schema.org/draft-04/schema#",
>>>> "title": "is the Body property well formed",
>>>> "description": "Does the body property contain either a URI, a JSON
>>>> Object that is a textual body, or a JSON Object that is a specific
>>>> property",
>>>> "compareWith": "or",
>>>> "assertions": [
>>>> ...
>>>> ]
>>>> }
>>>>
>>>> You get the idea. I can probably create a working example. However,
>>>> this is a pretty complicated case. Maybe we could start with something a
>>>> little simpler to demonstrate that it does what you want and then work our
>>>> way up to this?
>>>>
>>>> [1]
>>>> https://github.com/w3c/web-platform-tests/blob/master/annotation-model/CONTRIBUTING.md
>>>>
>>>> On Wed, Jul 13, 2016 at 10:21 AM, Robert Sanderson <azaroth42@gmail.com
>>>> > wrote:
>>>>
>>>>> Thanks Shane, I didn't find the OR syntax for the test (as opposed to
>>>>> within the schema itself).
>>>>>
>>>>> Could you (or someone) give an example of how the structured tests
>>>>> might work, as I must have missed it in the docs and current set of tests?
>>>>> In English, what I want to do is:
>>>>>
>>>>> * Test whether there's a body property or not
>>>>> * If there is, test if it's a JSON string.
>>>>> * If it is, test that it's a URI
>>>>> * If there is, test if it's a JSON object.
>>>>> * If it is, test if it's a TextualBody
>>>>> * If it is, test whether there's a value property or not
>>>>> * If there is, test that it's a string
>>>>> * ...
>>>>> * If it is, test if it's a SpecificResource
>>>>> * ...
>>>>> * Test whether there's a bodyValue property or not
>>>>> * If there is, test if it's a JSON string
>>>>> * Otherwise raise a warning that there's no body
>>>>>
>>>>> Where each of those is a separate schema, so they can be reused (e.g.
>>>>> value is used on many sorts of resources)
>>>>>
>>>>> Many thanks!
>>>>>
>>>>> Rob
>>>>>
>>>>>
>>>>> On Tue, Jul 12, 2016 at 4:13 PM, Shane McCarron <shane@spec-ops.io>
>>>>> wrote:
>>>>>
>>>>>> Umm... I am not clear what problem you are trying to solve.
>>>>>> Regardless, you can do what you express with the current syntax (which is
>>>>>> not a manifest). Each .test file expresses one or more assertions that
>>>>>> will be evaluated. You can have "or" clauses in the declarative syntax so
>>>>>> you can this OR this OR this OR this need to be true in order to satisfy
>>>>>> the requirements of the test.
>>>>>>
>>>>>> On Tue, Jul 12, 2016 at 6:01 PM, Robert Sanderson <
>>>>>> azaroth42@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> After playing around with the schemas over the weekend, with a view
>>>>>>> to integrating them into my server implementation to validate the incoming
>>>>>>> annotations, I ran into some issues:
>>>>>>>
>>>>>>> * The tests are designed for humans to read the error message, not
>>>>>>> for machines to process the results ... some tests are okay to fail
>>>>>>> validation, some aren't.
>>>>>>>
>>>>>>> * There doesn't seem to be a way to descend into the referenced
>>>>>>> resources automatically. You need to run the specific resource tests
>>>>>>> against the specific resource by hand.
>>>>>>>
>>>>>>> * The processing patterns of the single with break or skip seems
>>>>>>> like it could be extended ...
>>>>>>>
>>>>>>>
>>>>>>> So what do people think about the following, if it's not too late to
>>>>>>> change things:
>>>>>>>
>>>>>>> * Continue with atomic tests for presence of a property, and then a
>>>>>>> separate one for the value of it
>>>>>>> * But do that in the framework testing "manifest" by testing for
>>>>>>> failure/success of the validation.
>>>>>>>
>>>>>>> For example, an automated system could descend into a
>>>>>>> SpecificResource as the body by:
>>>>>>>
>>>>>>> * Test that body exists
>>>>>>> -- OnFail: Warn (body is a SHOULD)
>>>>>>> -- OnSuccess:
>>>>>>> * Determine type of body
>>>>>>> -- uri? OnSuccess: Test URI-ness & goto next set of tests
>>>>>>> OnFail: SpecificResource?
>>>>>>> OnSuccess: Descend into
>>>>>>> SpecificResource tests
>>>>>>> OnFail: TextualBody?
>>>>>>>
>>>>>>> And so forth.
>>>>>>> The success/fail would be by the same $ref approach as the schema
>>>>>>> includes, and offset from the schema itself, so it can be reused in
>>>>>>> different parts of the overall set of tests.
>>>>>>>
>>>>>>> This would let us compose features at whatever level we think is
>>>>>>> appropriate for reporting, and give a good validation suite for the model
>>>>>>> that can be used completely programmatically by implementing the
>>>>>>> success/fail runner.
>>>>>>> [I did this runner already in python, it's a pretty easy piece of
>>>>>>> code, as one might expect]
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Rob
>>>>>>>
>>>>>>> --
>>>>>>> Rob Sanderson
>>>>>>> Semantic Architect
>>>>>>> The Getty Trust
>>>>>>> Los Angeles, CA 90049
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shane McCarron
>>>>>> Projects Manager, Spec-Ops
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rob Sanderson
>>>>> Semantic Architect
>>>>> The Getty Trust
>>>>> Los Angeles, CA 90049
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Shane McCarron
>>>> Projects Manager, Spec-Ops
>>>>
>>>
>>>
>>>
>>> --
>>> Shane McCarron
>>> Projects Manager, Spec-Ops
>>>
>>
>>
>>
>> --
>> Rob Sanderson
>> Semantic Architect
>> The Getty Trust
>> Los Angeles, CA 90049
>>
>
>
>
> --
> Shane McCarron
> Projects Manager, Spec-Ops
>
--
Shane McCarron
Projects Manager, Spec-Ops
Received on Thursday, 14 July 2016 13:00:09 UTC