trying to enforce IRI syntax

On Fri, 2012-05-18 at 14:03 +0100, Richard Cyganiak wrote:
> 1) I will formally object to any notion of fixing-up-data-by-trying-to-guess-their-intent that is applied only to a single RDF syntax.
> 2) A general fixing-up-data-by-trying-to-guess-their-intent framework for all of RDF is out of scope for this WG.
> 3) Turtle validator. Think about it.

Hmmm.  What do other RDF syntaxes do with bad-syntax IRIs?

Quick check of the W3C RDF/XML validator show it doesn't even warn about
an IRI with "|" in it.   Nor does rapper (v 2.0.0), so it's not just
older parsers.  The 2010 RDFa distiller is also fine with this IRI.  The
2012 one is down right now.

The RDF/XML spec seems silent on it, deferring to RDF Concepts, which I
read as allowing it [1].   Specifically, it says the string must conform
to the URI character syntax AFTER disallowed characters (like "|") are
percent-encoded.   Possibly I'm reading that wrong.

Oh, I see in your new RDF Concepts draft you have a note about how this
changed.   So "|" was allowed in 2004 in RDF URIReference terms, but
wont be allowed in RDF 1.1 IRI terms?    Ouch.  


 - Will RDF/XML and RDFa parsers be required to reject documents
containing bad-syntax IRIs, eg containing a "|"?    (That does not seem
viable.  I doubt anyone would support that.)

 - so why should Turtle parsers have to do that?

Actually -- thinking about it some more, Turtle doesn't even reject IRIs
like this, either.  It just makes you type them in using the \u syntax!
The spec right now allows an IRI containing control characters, etc.   <
\u00> is a valid Turtle term.

So, given what I'm seeing so far, my proposal is:

1.  Simplify the Turtle grammar, making the IRIREF production be just:
"<" [^>]* ">".  Also simplify PNAME, etc, in corresponding ways.

2.  Somewhere, probably in 6.2, say IRIs (from both IRIREF and PNAME
productions, after prefix and base expansion and resolution) SHOULD be
checked against the IRI syntax rules and SHOULD NOT be emitted by the
parser or generated by the serializer.   This text should be coordinated
with rdf-concepts, factoring out stuff that applies to other RDF
syntaxes as well.  (What's there now [2], especially the last Note, is
good; the question is who is responsible for enforcing this stuff, and
how does it affect whether something is a conformant turtle serializer
or parser.)

     -- Sandro


> Best,
> Richard
> On 18 May 2012, at 13:08, Sandro Hawke wrote:
> > On Fri, 2012-05-18 at 12:22 +0100, Steve Harris wrote:
> >> Yes, we've actually had this issue in practice (lots of web URLs are not legal URIs), you want to find that out as early as possible that you have a problem. For us that's when PUTing Turtle to a RDF store.
> >> 
> >> - Steve
> >> 
> >> On 2012-05-18, at 11:20, Richard Cyganiak wrote:
> >> 
> >>> Sandro,
> >>> 
> >>> -1 to a “loose Turtle”.
> >>> 
> >>> If a conforming Turtle parser were allowed to accept a document containing <|b>, then what next? This is not a valid IRI. So it is not allowed in an RDF graph. A Turtle parser is rarely a stand-alone system — it's a component in a larger system. Once the Turtle parser tries passing on the pseudo-IRI to the next component, then a number of things can happen:
> >>> 
> >>> The next component might reject it outright.
> >>> 
> >>> Or the next component accepts it and stores the pseudo-IRI. Then the user can do their thing. Then when the user tries to save their work, the serializer checks IRIs and rejects it, taking down the app with an error message. (This is Jena's default behaviour, or at least was the last time I checked.)
> >>> 
> >>> Or maybe the entire system works, except that now we have a situation where certain RDF “graphs” can be loaded and saved in Turtle but not in other syntaxes. This will cause major headaches for users, who will end up messing around with format converters in order to get broken data into a format that doesn't complain about the data being broken.
> >>> 
> >>> Or maybe the system accepts the IRI and puts it into its store, but then you can't delete it from the store any more because the SPARQL Update part of the system is stricter and rejects DELETE DATA commands containing broken IRIs.
> >>> 
> >>> Given the complexity of RDF-bases systems, and the many interacting components and specifications involved, this kind of error handling cannot be introduced for a single syntax. It has to be done centrally so that all involved components and specifications can behave in a consistent way. Defining algorithms for error recovery for broken RDF data may well be a good idea, but I don't think this should be part of an 1.1 update to RDF, and I don't think we are chartered to do it.
> > 
> > Yeah, speaking as a a standards person, I absolutely agree with you.
> > 
> > But I put on my implementer's hat this week and had two problems:
> > 
> > 1.  The regexps seemed to be breaking various tools.  But it's very hard
> > to tell if it's the regexps or the tools, because of how big they are
> > (and I'm using Javascript which doesn't allow spaces or comments in
> > regexps).   For example, here's the regexp generated from our grammar
> > for IRIREF, with all the chars that I was particularly worried about
> > done as hex escapes:
> > 
> > /^(<([^\x00-\x20<>\x5c\x22\x7b\x7d\x7c^`\x5c]|((\\x5cu([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f]))|(\\x5cU([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f])([0-9]|[A-F]|[a-f]))))*>)
> > 
> > Actually, looking at it now, I see two errors in it, caused by how jison
> > composes named regexps containing backslashes.
> > 
> > So, obviously I could do this more carefully, writing my own tool to do
> > macro-composition of regexps, trying to find something better, or fixing
> > jison.    But still, ouch.
> > 
> > 2.  If/when I ship my software, it's pretty clear I'm going to get bug
> > reports from users about these syntax errors in files that look fine to
> > them, and/or they don't control but want to be able to read anyway.  I
> > can tell them, "sorry, check the spec", but, ... another pain point.
> > 
> >> On Fri, 2012-05-18 at 12:22 +0100, Steve Harris wrote:
> >> Yes, we've actually had this issue in practice (lots of web URLs are  
> >> not legal URIs), you want to find that out as early as possible that  
> >> you have a problem. For us that's when PUTing Turtle to a RDF store.
> > 
> > So, how about this:
> > 
> > 1.  We make the Turtle grammar much simpler on this.  For example:
> >        IRIREF ::= /^<[^ \t\n\r>]*>/ 
> > 
> > 2.  We say that Turtle parsers MUST check IRIREFs for conforming to the
> > IRI spec -- maybe we give the regexp for that, or maybe we just refer
> > them to the right RFCs.  If an IRI fails the check, the parser MAY
> > transform the IRI into one which would pass the check, using percent
> > encoding.  The parser MUST not emit RDF triples containing IRIs which do
> > not syntactically conform to IRIand URI RFCs.    (I'd also be fine with
> > SHOULD NOT; people might, for instance, know the next stage upstream is
> > going to do this anyway.)
> > 
> > 3.  We say the same sort of thing about Turtle generators; they MAY do
> > percent-encoding if handed bad stuff; they MUST NOT emit bad IRIs.
> > 
> > So, we'd be permitting error recovery -- but very well-defined error
> > recovery.    (And something the browsers do all the time; you can give
> > them anything for a URL and they'll percent-encode it as needed.)
> > 
> > We'd also be using a standard bit of code -- the IRI checker isn't in
> > any way Turtle-specific -- instead of making the Turtle lexer super
> > complicated.
> > 
> >   -- Sandro
> > 
> >>> Best,
> >>> Richard
> >>> 
> >>> 
> >>> On 17 May 2012, at 21:35, Sandro Hawke wrote:
> >>> 
> >>>> What should/may/must a Turtle parser do with a turtle document like
> >>>> this:
> >>>> 
> >>>> <> <> <|b>.
> >>>> 
> >>>> By the grammar, this is not a Turtle document, because of the '|'
> >>>> character in a URI.   I don't think, however, that people writing Turtle
> >>>> parsers will want to enforce this.  If they come across some Turtle
> >>>> document that's got a URI like this -- they can still parse it just
> >>>> fine, so they probably will.
> >>>> 
> >>>> The language tokens like IRIREF and PNAME are defined in the grammar
> >>>> with these vast regexps (if you macro-expand what's there, now), but
> >>>> actually much simpler ones will produce the same result in practice --
> >>>> they'll just tolerate some files that are not, strictly-speaking,
> >>>> Turtle.  (I'm pretty sure -- maybe there are some corner cases with
> >>>> missing whitespace where these regexps will give you a different result
> >>>> than something more like any-character-up-until-a-delimiter.
> >>>> 
> >>>> I'm not sure anything has to change, but I think at very least the
> >>>> conformance clause should be clear about whether it's okay to accept a
> >>>> turtle document like my example above.
> >>>> 
> >>>> It might be nice to have "strict" and "loose" parsers, especially if we
> >>>> can define loose parsers in a way that makes them simpler to implement,
> >>>> run faster, and never parse anything differently from a strict parser.
> >>>> 
> >>>> Of course, then I'm not quite sure the point of the strict parsers.
> >>>> 
> >>>> -- Sandro
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>> 
> >>> 
> >> 
> > 
> > 
> > 

Received on Friday, 18 May 2012 15:41:05 UTC