- From: Tim Cole <t-cole3@illinois.edu>
- Date: Sat, 11 Jun 2016 12:40:22 -0500
- To: Discussions about Test Development <testdev@lists.spec-ops.io>
- CC: W3C Public Annotation List <public-annotation@w3.org>
- Message-ID: <4ea72cm2si93bph72s23myb9.1465666689522@email.android.com>
Fine with me, thanks Shane.
FYI, as indicated in the draft spread sheet i put in google docs, we need one more folder or sub folder for recognizing and vetting Agent objects. These are used across multiple objects, so the folder could go as a subfolder to /common, if we retain that as a top level folder, or could go as a top level folder on its own. Either way fine with me.
Traveling today, but will add more schemas tommotw.
Tim Cole
Univ of Illinois at UC
Sent from my phone. Apologies for typos.
<div>-------- Original message --------</div><div>From: Shane McCarron <shane@spec-ops.io> </div><div>Date:06/11/2016 11:23 (GMT-06:00) </div><div>To: Discussions about Test Development <testdev@lists.spec-ops.io> </div><div>Cc: W3C Public Annotation List <public-annotation@w3.org> </div><div>Subject: Re: [Testdev] [model-test] folder tree -- was: referencing external schema </div><div>
</div>I am going to add these top level folders today so that I can continue to make progress. This may require moving some things around. I hope that doesn't disrupt anything too much.
On Thu, Jun 9, 2016 at 12:44 PM, Shane McCarron <shane@spec-ops.io> wrote:
Typically the section numbers are not included in the directory name. So I would do something like:
/annotations
/bodiesTargets
/specificResources
/specificResources/selectors
/specificResources/states
/collections
In other words, structure it like the hierarchy of the spec itself. Does that make sense? And yes, go ahead and add them.
And if you think things are better in subfolders than in common, go for that. I assumed things would get used a lot and so we would need a "common" folder. If that is NOT the case, should we change the rule on relative "assertionFile" paths to be relative to the current directory rather than to test suite top?
On Thu, Jun 9, 2016 at 11:47 AM, Timothy Cole <t-cole3@illinois.edu> wrote:
Okay, so obvious folders for the annotation data model would be:
/common
/definitions (as we've discussed)
/3.1annotations
/3.2bodiesTargets
/4.specificResources
/4.2selectors
/4.3states
/5.collections
We can leave the numbers off if preferred, but obviously these tie to specific sections of the model spec. And of course happy to follow wpt preferred naming patterns if you let us know what those are.
Everything currently in /common would need to be redistributed to 3.1annotations or 3.2bodiesTargets; almost none of the schema currently in this folder (with possibly exception of id.json and idFormat.json) would be re-used for checking implementation of anything other than the Annotation object itself, so unlikely to appear in multiple tests.
Note also that to avoid making too many folders I'm suggesting that the 4.specificResources folder contain schema for all parts of Section 4 not specific to selectors and states (i.e., would include schema for checking and validating implementation of purpose, styles, renderedVia and scope as well as for identifying and checking implementations of SpecificResource generally).
Could also add a 3.3otherProperties folder, but this folder would contain schema that would be used across multiple sections, so would it be better to use /common for these schema or to create the additional folder?
Does this folder list make sense in context of wpt best practices?
Let me know preferred naming convention, and whether okay to add these half-dozen folders as siblings to /common.
Thanks,
Tim Cole
From: Testdev [mailto:testdev-bounces@lists.spec-ops.io] On Behalf Of Shane McCarron
Sent: Thursday, June 09, 2016 10:59 AM
To: Discussions about Test Development <testdev@lists.spec-ops.io>
Cc: W3C Public Annotation List <public-annotation@w3.org>
Subject: Re: [Testdev] [model-test] referencing external schema
Typically WPT folder trees are designed to mirror the structure of the spec being tested. That way it is easy to correlate the assertions with the content of the spec. Beyond this, there are sometimes top level folders like "tools" or "common". I think it is important that we stick with that sort of structure to help with adoption by the wpt community.
On Wed, Jun 8, 2016 at 5:37 PM, Timothy Cole <t-cole3@illinois.edu> wrote:
I think either approach should work, but I've had some issues even with referencing absolute URIs . The ajv import approach might be a little easier and more reliable? Regardless, now that the patterns are becoming clear, copying definitions to multiple schemas not all that onerous. You can always include definitions in a schema and not reference… Obviously the big issue is when you want to revise a definition (or upgrade to schema version 5). Then having to go back and fix in multiple files would indeed be an issue. So hopefully one of your approaches will work.
To give some concrete examples of definitions reuse (and maybe make it easier to test your methods?), I've been putting schemas mostly here, https://github.com/w3c/web-annotation-tests/tree/master/common. Take a look at bodyResource.json and targetResource.json. These schemas are designed to check that body/target resource(s) is(are all) instance(s) of classes allowed to be a body or target. Both schemas use essentially the same definitions (the bodyResource.json has a textualBody definition not used in the targetResource.json). There's also one schema I just put in a sibling folder /TextualBody/textualBody.json. This schema is designed to return valid if any body of an Annotation is a TextualBody and reuses one of the common definitions and adds an additional definition that will be used in multiple schemas. This schema tells us if the TextualBody feature was implemented in an annotation and determines if further schemas should be run to identify which sub-features of TextualBody were implemented and whether requirements re TextualBody and its sub-features were met.
Which raises the question of folder design to handle all the schemas that are in the pipeline. Regarding top-level folders, i.e., siblings of /common, I've been reworking Rob's original features spreadsheet. I think we will need several siblings of /common, not only for definitions, but also for selected child objects of an Annotation that themselves can have a number of unique keys and their own child objects (e.g., SpecificResource, TextualBody as discussed above, etc.). I think this is best way to manage what is going to eventually be a relatively large number of schemas. If you want to suggest a different folder hierarchy approach please do – but can't all go in /common (which I think you've already said).
For example, there are 17 keys in the model that can be used on an Annotation (16 at any one time, 2 are mutually exclusive). About 7 or 8 of these keys (e.g., id, type, and life-cycle keys) can appear on other Annotation child objects, so relevant schemas will be re-used. In a few cases the constraints on a key varies according to it's parent. Since some keys are should or may, but then if used have must constraints, I'm mostly following a pattern of 1 schema to determine if the feature is present, and a second schema to determine if the feature was implemented correctly. Thus, id is a 'should' key on an Annotation, but if present id must be singular (array not allowed) and its value must be of format uri. So if id is not present the invalid message (first schema) is "Warning: The Annotation is not identified using the id key (Section 3.1)," but if id is present and not a uri, then the invalid message (2nd schema) is: "Error: The Annotation identifier ('id') is not a string of format uri (Section 3.1)." If the first schema returns invalid, the second schema is skipped. You get the idea. So in the /common folder I think we will end up with about 35 to 40 schemas. Hence the need for sibling folders I've added (or alternatively for child folders).
I don't know as we'll get this far into the weeds on the call Friday, but wanted to flag folder management as a topic of discussion on email at least.
I will post a message tomorrow with a link to an incomplete version of revised spread sheet and about schema granularity to get this conversation up and going again. Not necessarily critical for the testing plan we need for CR, but to actually get to point where we can start running test, we need to figure this stuff out.
-Tim Cole
From: Testdev [mailto:testdev-bounces@lists.spec-ops.io] On Behalf Of Shane McCarron
Sent: Wednesday, June 08, 2016 4:30 PM
To: Discussions about Test Development <testdev@lists.spec-ops.io>
Cc: W3C Public Annotation List <public-annotation@w3.org>
Subject: Re: [Testdev] [model-test] referencing external schema
Okay. We have two strategies in mind:
1) have a top level definitions folder, then use absolute URIs (e.g., /annotation-model/definitions/my_def.json)
2) have that top level folder, and have ajv automatically import all of the .json files in that folder during initialization.
Would either of those work for you Tim?
On Wed, Jun 8, 2016 at 2:50 PM, Timothy Cole <t-cole3@illinois.edu> wrote:
Yes. I actually had created the definitions folder before I realized relative addressing wasn't working. I'm in the process of creating some better examples of re-useable definitions – the samples I included in my email are out of date and will be replaced.
Thanks,
Tim Cole
From: Testdev [mailto:testdev-bounces@lists.spec-ops.io] On Behalf Of Benjamin Young
Sent: Wednesday, June 08, 2016 2:26 PM
To: Discussions about Test Development <testdev@lists.spec-ops.io>
Cc: W3C Public Annotation List <public-annotation@w3.org>
Subject: Re: [Testdev] [model-test] referencing external schema
Hey Tim,
Sorry for the wait. Shane and I just chatted about this $ref thing and I think we can make it work in WPT. It’s mostly a question of what the relative path expectations of `ajv` tucked inside of WPT might be.
It looks like we may also be able to hack around the defaults (if need be) using wptserve’s Pipes system:
https://wptserve.readthedocs.io/en/latest/pipes.html
What we’re thinking right now is taking the JSON objects you now have in the `definitions` object (which you pasted below) and putting those inside of a `definitions` folder and then working through the relative path details with WPT.
Would that serve as a solution for what you’re attempting here?
Cheers!
Benjamin
From: Testdev [mailto:testdev-bounces@lists.spec-ops.io] On Behalf Of Shane McCarron
Sent: Wednesday, June 8, 2016 10:46 AM
To: Discussions about Test Development <testdev@lists.spec-ops.io>
Cc: W3C Public Annotation List <public-annotation@w3.org>
Subject: Re: [Testdev] [model-test] referencing external schema
Did you get any traction on this Tim?
On Thu, Jun 2, 2016 at 10:19 AM, Timothy Cole <t-cole3@illinois.edu> wrote:
We are currently developing json schemas to help test for implementations of SpecificResource features of the data model. I've uploaded 4 of the schemas currently being developed to https://github.com/w3c/web-annotation-tests/tree/master/SpecificResource (these are first drafts, not quite ready, still being debugged and refined). All 4 schemas share a definitions section containing 2 sub-schemas (appended below). These are easy to reference internally where needed, but it would be even more convenient to reference them as external schemas. This should be possible using $ref, but I've had difficulties getting this approach to work with my local ajv instance, especially with regard to relative file locations.
Is anyone familiar enough with how $ref works within the Web Platform Testing framework's implementation of ajv to advise?
Below is the current way the sub-schemas are embedded in each primary schema. Obviously this works, but it means repeating the same sub-schemas in multiple files, which is obviously less efficient than if these sub-schemas could be referenced as individual files.
Thanks,
Tim Cole
{
"definitions": {
"SpecificResource": {
"type": "object",
"properties": {
"selector": {},
"state": {},
"styleClass": {},
"renderedVia": {},
"scope": {},
"purpose": {}
},
"anyOf": [
{"required": ["selector"]},
{"required": ["state"]},
{"required": ["styleClass"]},
{"required": ["renderedVia"]},
{"required": ["scope"]},
{
"required": ["purpose"],
"not": {"required": ["value"]}
}
]
},
"SpecificResType" : {
"type": "object",
"properties": {
"type": {
"oneOf": [
{"type": "string",
"enum": ["SpecificResource"]},
{"type": "array",
"items": [{ "type": "string",
"enum": ["SpecificResource"] }]
} ]
}
},
"required": ["type"]
}
}
}
_______________________________________________
Testdev mailing list
Testdev@lists.spec-ops.io
http://lists.spec-ops.io/listinfo.cgi/testdev-spec-ops.io
--
Shane McCarron
Projects Manager, Spec-Ops
_______________________________________________
Testdev mailing list
Testdev@lists.spec-ops.io
http://lists.spec-ops.io/listinfo.cgi/testdev-spec-ops.io
--
Shane McCarron
Projects Manager, Spec-Ops
_______________________________________________
Testdev mailing list
Testdev@lists.spec-ops.io
http://lists.spec-ops.io/listinfo.cgi/testdev-spec-ops.io
--
Shane McCarron
Projects Manager, Spec-Ops
_______________________________________________
Testdev mailing list
Testdev@lists.spec-ops.io
http://lists.spec-ops.io/listinfo.cgi/testdev-spec-ops.io
--
Shane McCarron
Projects Manager, Spec-Ops
--
Shane McCarron
Projects Manager, Spec-Ops
Received on Saturday, 11 June 2016 17:40:59 UTC