Re: JSON-LD and JSON-Schema linkage

Henry,

I do appreciate the context and thought you (and others) have put into the
review of the current standard.

I'm going to go with using $schema, and I'll post a link to the demo site
once I'm done so the list can see what I'm talking about in action.

Keep up the good work and I'll be looking to post some comments on the new
draft!

-TJ

On Wed, Oct 25, 2017 at 12:03 AM, Henry Andrews <henry@cloudflare.com>
wrote:

> Techinically "$schema" is for use in schemas referencing meta-schemas
> (meaning that a meta-schema references itself that way).  However, I have
> seen other users put "$schema" in instance JSON for this purpose.  It is
> definitely allowable as JSON Schema avoids constraining the JSON structure
> as a pre-condition for use.
>
> Regarding Swagger (and I know this is drifting off-topic, we can take this
> offlist if folks prefer), OpenAPI 3.x is *almost* compatible with JSON
> Schema validation (the draft-04 / draft-05 versions, which are effectively
> the same).  For some reason they insisted on refusing to support things
> like:
>
> "type": ["integer", "null"]
>
> and instead only allow single-value types and added an incompatible
> boolean flag, "nullable".  I and a few others are trying to convince them
> to support two-element type arrays where one element is "null" for
> compatibility purposes, even if they don't want to fully support type
> arrays (which is fine- they're rarely used aside from the type-plus-null
> use case anyway).  It would be really great to get that one remaining
> conflict out so that everything that is supported in OpenAPI with JSON
> Schema uses the JSON Schema syntax.
>
> Hyper-schema is really a complementary technology emphasizing hypermedia,
> links, and HATEOAS, where Swagger is a static API description format
> emphasizing .  exact request/response syntax regardless of HTTP semantics.
> That's far too long and tangential a topic for this mailing list but I do
> get into the difference a bit in hyper-schema draft-07.
>
> thanks,
> -henry
>
>
> On Tue, Oct 24, 2017 at 8:14 PM, TJ Koury <tjkoury@gmail.com> wrote:
>
>> Henry,
>>
>> I most definitely agree with your thoughts, and I think allowing document
>> fragments or a $schema property to be interpreted as JSON Schema is a huge
>> step in the right direction.
>>
>> Honestly I haven't used the validation or hyper schemas due to the lack
>> of documented interop with Swagger et al, glad to see you think that is an
>> issue as well.
>>
>> For now I'll just use the $schema property, but I will try to engage
>> going forward and provide code examples too.
>>
>> -TJ
>>
>> On Oct 24, 2017 10:44 PM, "Henry Andrews" <henry@cloudflare.com> wrote:
>>
>> TJ,
>>
>> The HTTP header stuff (either the Link header, or using a media type
>> parameter in Accept on a request, and getting one back in Content-Type on
>> the response, or sending one in Content-Type on a request) is really
>> designed for a simple instance document + schemas paradigm (1-to-many, as
>> you can have multiple links or multiple schema URIs in the media type
>> paramter).
>>
>> What it really does not address well is the case where JSON Schema is
>> applied to *part* of a document, or where *part* of a document is JSON
>> Schema, but the entire document is not (at least as we define JSON Schema
>> right now).  These are problems that I want to focus on for draft-08, which
>> will have re-use as its theme (although I am intentionally not filing
>> further issues on them yet to keep folks focused on the draft-07
>> discussion).
>>
>> I am trying to build a case for JSON Schema as a media type supporting
>> many vocabularies.  The terms in the vocabularies are either assertions,
>> annotations, or both (these terms are defined in the draft-07 Validation
>> document- they were not called out before).  This is sort of true now:
>> Validation and Hyper-Schema are both described as "vocabularies", although
>> their relationship with each other and the core specification is somewhat
>> muddled.  This lack of clarity makes it hard to define extended,
>> restricted, or orthogonal vocabularies.
>>
>> In particular, an extended or restricted vocabulary can only be indicated
>> by a single "$schema" URI, which makes it impossible to detect the
>> underlying vocabulary even if an implementation would be able to make use
>> of that portion of the document.  This would enable graceful degradation.
>> For example, a validator can still use a hyper-schema, it just ignores the
>> "base" and "links" keywords and everything works just fine.  However, this
>> only "works" because implementations hardcode that hyper-schemas are also
>> usable as validation.  It would not work with a custom extension to
>> validation, which I would like to fix (I am getting pushback on this, so if
>> anyone would like this flexibility, I could use your support).
>>
>> I want this sort of flexibility because a major use case for JSON Schema
>> is not whole documents but partial documents, on both the schema and
>> instance side.
>>
>> API description formats like OpenAPI (a.k.a. Swagger) use JSON Schema (or
>> a subset of it) in parts of a JSON or YAML file.  It would be great to be
>> able to consider such a file to itself be a JSON Schema, mostly with a
>> custom vocabulary, but one that includes a standard vocabulary in certain
>> spots so that JSON Schema tools could recognize and work with the whole
>> thing in an interoperable mannter.
>>
>> Even more relevant to JSON-LD, the W3C's "Web of Things" group's Thing
>> Description format is a mostly-JSON-LD-based document that uses JSON Schema
>> in a few spots for structure descriptions.  This sort of mixture seems
>> useful and desirable.
>>
>> And now as you mention, there may be data that we want to describe with
>> JSON Schema (either as its own document or part of something larger)
>> embedded within documents that we do not wish to fully describe, such as
>> JSON-LD within an HTML document.  I had not previously considered this
>> case, but it fits well with the overall direction I want to consider in
>> draft-08.
>>
>> Basically, I want to make all of these things first-class use cases for
>> JSON Schema, such that tools that support "JSON Schema" (whatever that ends
>> up formally meaning) can work with such situations.
>>
>> Those of us who work with hyper-schema, web UI annotation, code
>> generation, document generation, and other non-validation uses tend to
>> support modular re-usability as I have sketched it out above.  Some of
>> those who focus on validation only have opposed it (in at least one
>> prominent case, vehemently).  So I am hoping to build support for a
>> flexible definition of JSON Schema, as opposed to the "JSON Schema is
>> validation with incidental other stuff hanging off of it" viewpoint.
>>
>> This debate will probably kick off towards the end of this month,
>> although of course folks are welcome to file issues and/or comment before
>> then.  This is the issue that is primarily tracking being able to detect
>> the base vocabulary of an extended system: https://github.com/json-schema
>> -org/json-schema-spec/issues/314  The other things do not have issues
>> yet (I'll file them when draft-07 is out the door later this month)
>>
>> thanks,
>> -henry
>>
>> PS: For those interested in the road map (which is not endorsed by anyone
>> but me right now):
>>
>> draft-07 was about hypermedia (including a top-to-bottom rewrite of JSON
>> Hyper-Schema)
>> draft-08 will be about re-use
>> draft-09 will *probably* be about whether "$data" and similar concepts
>> should be added
>> after that, I hope we can get JSON Schema, at least core and validation,
>> adopted by an IETF working group or otherwise start bringing the process to
>> some sort of resolution.
>>
>>
>> On Tue, Oct 24, 2017 at 6:25 PM, TJ Koury <tjkoury@gmail.com> wrote:
>>
>>> Henry,
>>>
>>> Thanks for the quick reply.  If you're looking for inputs, and I can
>>> submit something formal if I'm not too off base here, I'd like to see
>>> something along the lines of providing an schema property as a child of a
>>> context node.
>>>
>>> The goal would be the ability to make a one to many relationship with a
>>> linked json being able to be associated with multiple schemas; in my case,
>>> a NIEM entity instance being described for multiple persistence engines.
>>>
>>> I'm of the opinion that using HTTP headers for schema references is not
>>> a great option for several reasons, such as not being able to accurately
>>> describe an HTML document with an embedded JSON-LD tag, and also forcing a
>>> consumer to persist data pertaining to the  model outside of the standard.
>>> In my specific use case, I'd like the schemas to be canonical when
>>> referencing authoritative data sources, and not be an implemention detail
>>> at time of data request.
>>>
>>>
>>> On Oct 24, 2017 8:41 PM, "Henry Andrews" <henry@cloudflare.com> wrote:
>>>
>>> Hi TJ,
>>>   I'm currently the most active editor of the JSON Schema specification,
>>> and this has been a recent topic of discussion for the forthcoming (no
>>> later than Nov. 20th, barring unexpected problems) draft-07.
>>>
>>>   I see you are using draft-04 for this.  In that draft (and up through
>>> draft-06) the recommendations were:
>>>
>>> * Use HTTP link headers if relevant:  "profile" as an identifier,
>>> "describedBy" as a locator (most of the time they would be the same)
>>> * Use "profile" as a media type parameter
>>>
>>> However, per the author of the "profile" RFC this usage was never quite
>>> right (JSON-LD use it correctly, though).  And application/json does not
>>> support a profile media type parameter anyway.
>>>
>>>   In draft-07, we are (probably) proposing replacing "profile" with a
>>> newly proposed "schema" link relation type and/or media type parameter.
>>> JSON-LD could opt to support the media type parameter with
>>> application/ld+json, or just use a "schema" link.  Whether this is correct,
>>> or sufficient, or if there is a better approach, is something we really
>>> hope to get feedback on with draft-07.  It is the main unresolved concern
>>> with the core spec, as far as I know.  And I still find the
>>> "describedBy"-as-locator part a bit odd, personally.
>>>
>>>   If anyone wants to see a preview of draft-07, you can find it here:
>>> http://json-schema.org/work-in-progress
>>>
>>> thanks,
>>> -henry
>>>
>>>
>>> On Tue, Oct 24, 2017 at 3:55 PM, TJ Koury <tjkoury@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> ALCON,
>>>>
>>>> This has been asked before several times, but the answers always seem
>>>> to get muddled down and lost in semantics (which is what we’re doing here,
>>>> so I get it….).
>>>>
>>>> For reference:
>>>>
>>>> https://github.com/json-ld/json-ld.org/commit/019de59e296c39
>>>> d7b5c0298d49d95b99fceb294a
>>>>
>>>> So there is a JSON-Schema document to validate ANY JSON-LD document
>>>> against to make sure that it’s a valid JSON-LD document.  Awesome!
>>>>
>>>> Now, if I have a http://schema.org/Person, and an associated
>>>> JSON-Schema document that defines the fields associated with Person, how do
>>>> I add a URL to the associated JSON-LD document to reference that schema?
>>>>
>>>> My specific use case is to generate JSON-LD documents from NIEM
>>>> instances, then create JSON-Schema documents for each persistence engine
>>>> (database, file system, etc) that will store the instances, and embed
>>>> within the JSON-Schema documents themselves the metadata required to create
>>>> the tables.  Basically an Schema->SQL engine (another ORM!), but entirely
>>>> based on the JSON-LD and JSON-Schema specs.
>>>>
>>>> -TJ
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>    -
>>>
>>>    *Henry Andrews*  |  Systems Engineer
>>>    henry@cloudflare.com
>>>    <https://www.cloudflare.com/>
>>>
>>>    1 888 99 FLARE  |  www.cloudflare.com
>>>    -
>>>
>>>
>>>
>>>
>>
>>
>> --
>>
>>    -
>>
>>    *Henry Andrews*  |  Systems Engineer
>>    henry@cloudflare.com
>>    <https://www.cloudflare.com/>
>>
>>    1 888 99 FLARE  |  www.cloudflare.com
>>    -
>>
>>
>>
>
>
> --
>
>    -
>
>    *Henry Andrews*  |  Systems Engineer
>    henry@cloudflare.com
>    <https://www.cloudflare.com/>
>
>    1 888 99 FLARE  |  www.cloudflare.com
>    -
>
>