Re: progress on schemas for model testing

Comments inline:

On Thu, Jul 28, 2016 at 1:49 AM, Tim Cole <t-cole3@illinois.edu> wrote:

> Following up on last 2 Working Group calls, we've re-organized and
> consolidated the schemas in the
> https://github.com/w3c/web-annotation-tests repository. We do not yet
> have repository fully populated with all required schema, but we have made
> significant progress with the new approach and will continue if the
> consensus of the WG is that we're now on the right track.
>
>
>
> The /definitions folder now contains 6 JSON files, each of which contains
> multiple sub-schemas that can be referenced directly by other schemas to
> facilitate testing and validation. Most of these files are not yet fully
> populated, and we may need additional files, but consolidating in a smaller
> number of files will facilitate management of these definitional schemas
> and will acilitate loading them as reference schemas. The files roughly
> align with sections of the model spec:
>
>
>
> Annotations.json - annotation-specific properties, checks and constraints
> like @context and "type": "Annotation", Section 3.1
>
>
>
> bodyTarget.json - properties, checks and constraints related to targets
> and/or bodies, Section 3.2.1 - 3.2.6
>
>
>
> choiceSet.json - properties, checks and constraints related to choice,
> composite, list, independents,, Section 3.2.7 - 3.2.8
>
>
>
> otherProperties.json - life cycle and other properties, checks and
> constraints used with annotations, bodies and targets , Section 3.3
>
>
>
> specificResource.json - properties, checks and constraints related to
> Specific Resources, Section 4.
>
>
>
> id.json - checks to validate id and strings of format uri
>
>
>
> Other folders in the repository contain higher level schemas that can be
> referenced from test scripts or that can be useful for validation. For
> example,  /annotations/3.1-annotationMustKeys.json contains schemas for
> reference by test scripts for the MUST requirements for Annotations - i.e.,
> @context, id that is URI, type, only one of body and bodyValue, target.
> This schema can also be used independent of the WPT test tools as a JSON
> Schema way to validate presence and correctness of mandatory Annotation
> keys.
>
>
>
> /bodiesTargets/3.2.1-6-BodyResources.json and
> /bodiesTargets/3.2.1-6-TargetResources.json contain schemas for detecting
> and validating (to an extent at least) optional feature implementation,
> differentiating between different kinds of bodies and targets and
> validating value correctness of mostly body/target optional key values.
> Again these 2 schemas, assuming the local availability of validating
> parsers like ajv-cli, can be used to locally validate correctness of
> optional keys - i.e., giving a valid result if feature is present and
> correct, or is not present at all, but giving an invalid result if the
> feature is present but is incorrect (e.g., if an object rather than a
> string or if value is not consistent with enumerated list of values). There
> are also some schema (with incorrect  expectedResult values I notice now)
> that can be used to report that a specific feature has been implemented..
>
>
>
> Please note that these schema are unfinished and not fully checked. You'll
> find errors but it’s a good start and hopefully makes a little more sense
> than what we had 2 weeks ago..
>
>
>
> One important question before proceeding much further.  - For the purposes
> of W3C conformance testing, how important is it to report a count of how
> many times within a single annotation a feature appears, or when multiple
> bodies or target are involved in an Annotation, do we need to know on which
> class of body or target a particular feature was used?  For example do we
> need to count the use of textDirection on embedded Textual Body separately
> from the use of textDirection on an External Web Resource? Or is it
> sufficient to simply report that textDirection feature was used on a body
> or bodies in a given annotation?
>

Hmm... I think we need to decide what a "feature" is for CR purposes.  For
example, textDirection does not feel like a feature to me.  It is just an
attribute of the annotation.  There are lots of attributes, right?  I think
that for CR the feature granularity should be at a higher level.  Do
annotation clients support the use of bodyValue?  of body? In body, which
body types are used?  That's enough.



>
>
> Otherwise please comment and/or contribute. You'll note that full naming
> convention is not fully in place yet. Test scripts still need to be
> organized and written.  Logic seems a bit non-intuitive, which may be the
> nature of how we're trying to use json schema; but it also may be just bad
> schema writing.
>
>
>
>
>
I don't understand what this means.  Sorry.  What is non-intuitive?  Is
there a way to provide some examples?

As to getting actual tests written... I had really hoped those would exist
by now.  Let's try to have a concrete plan by the end of today's call.


-- 
Shane McCarron
Projects Manager, Spec-Ops