Re: The xs:anyURI problem…

Dear Norm,

Your impressive summary leaves me in agreement that we should do something.

I wonder how many users actually need or want the old ('wrong') behavior.

For my projects, I'd be willing (happy) to make necessary changes, if I
find any to make - my mental model does not have me relying on this as a
feature, "where you’re using a relative URI and relying on it remaining
relative" (your observation 1), so wherever it lurks I'm happy to think of
it as a bug to be found.

Whatever we do about namespace-rename, this makes me lean towards a strict
approach of effectively deprecating and discouraging this. Especially if,
as it sounds like, this was simply a lapse that got by both you and Achim
more or less by accident, to be discovered by George under 'real world'
(ish) scenarios.

(I know about living with past mistakes but I prefer to live with the ones
that were intentional, to some definition of that term. And some mistakes
can be corrected. :-)

Error handling when URIs fail might also be enhanced/enhanceable with
diagnostic info such as relative-part and base URI.

This isn't a vote, just an inclination with more questions. If there's
anyone with a circumstantial (i.e., not a principled and theoretical)
reason for wanting 'not to fix it', that would be relevant info to me.

I'd also like to know if you, Achim, George and others feel (as I do) that
the stricter model is more robust and secure. (And as you say, one can
always pass a string or heck, even a function?)

Thanks and happy solstice, today is actually longer, in the 'more daylight'
sense --

Cheers, Wendell


On Wed, Dec 24, 2025 at 7:03 AM Norm Tovey-Walsh <ndw@nwalsh.com> wrote:

> [ This turned out to be a longer message than I expected, even when I
> expected it to be long. Sorry about that. ]
>
> Hello and happy seasonal greetings!
>
> It’s been six months and we haven’t resolved this issue. It feels like we
> should grasp the nettle and resolve it.
>
> To recap:
>
> Given a step, x:mystep, declared in /path/to/lib/mystep.xpl
>
> <p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
>                 xmlns:xs="http://www.w3.org/2001/XMLSchema"
>                 xmlns:x="http://example.com/ns/steps"
>                 name="main" version="3.1" type="x:mystep"
>                 exclude-inline-prefixes="#all">
>   <p:output port="result"/>
>   <p:option name="file" as="xs:anyURI"/>
>   <p:identity>
>     <p:with-input><doc>{$file}</doc></p:with-input>
>   </p:identity>
> </p:declare-step>
>
> And a pipeline in /path/to/pipeline.xpl:
>
> <p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
>                 xmlns:xs="http://www.w3.org/2001/XMLSchema"
>                 xmlns:x="http://example.com/ns/steps"
>                 name="main" version="3.1"
>                 exclude-inline-prefixes="#all">
>   <p:import href="lib/mystep.xpl"/>
>   <p:output port="result"/>
>
>   <x:mystep file="pipe.xpl"/>
> </p:declare-step>
>
> What should the output be? The answer hinges on how the xs:anyURI option
> “file” is handled.
>
> Today, in my implementation and Achim’s, the output is <doc>pipe.xpl</doc>.
>
> George Bina made the argument that this is wrong. The spec says of
> xs:anyURI options:
>
>    If [an] option value is specified using a syntactic shortcut, the base
> URI of the step element on which the shortcut attribute appears must be
> used [as the base URI to resolve against].
>
> With that interpretation, the output is
>
>   <doc>file:/path/to/pipe.xpl</doc>
>
> George’s argument is thorough and thoroughly supported by what the spec
> actually says. I think he’s right.
>
> Gerrit noted that the relevant prose in the specification goes on to say:
>
>   In general, whenever a relative URI appears in an xs:anyURI, its base
> URI is the base URI of the nearest ancestor element.
>
> That complicates things a bit, and makes it more likely that the change
> proposed will have unforseen consequences in existing pipelines, because it
> applies in more places than just options. Consider:
>
> <p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
>                 xmlns:xs="http://www.w3.org/2001/XMLSchema"
>                 xmlns:x="http://example.com/ns/steps"
>                 name="main" version="3.1"
>                 exclude-inline-prefixes="#all">
>   <p:import href="lib/mystep.xpl"/>
>   <p:output port="result"/>
>
>   <p:variable name="filename" as="xs:anyURI"
>               xml:base="/alternate/path/" select="'pipe.xpl'"/>
>
>   <x:mystep file="{$filename}"/>
> </p:declare-step>
>
> Here, curiously, MorganaXProc-III (1.6.8 which is what I happen to have
> installed at the moment) and XML Calabash give different results. Morgana
> gives
>
>   <doc>pipe.xpl</doc>
>
> which is, I admit, what I expected. But XML Calabash gives
>
>   <doc>file:/path/to/pipe.xpl</doc>
>
> This happens because after resolving the AVT to “pipe.xpl”, the expected
> type of the option is xs:anyURI, so it’s resolved against the current base
> URI.
>
> <aside>
> ============================================================
> In fact, the result from the first pipeline, <doc>pipe.xpl</doc>, is
> arguably incorrect. The correct result is also
> <doc>file:/path/to/pipe.xpl</doc>. The constant value is not being resolved
> against the base URI, but I think it should be. Let’s leave that to one
> side for the moment. Irrespective of which answer you think is correct,
> “pipe.xpl” and $filename should give the same result.
> ============================================================
> </aside>
>
> In any case case, the general rule for resolving base URIs says that the
> value of $filename should be file:/alternate/path/pipe.xpl.
>
> To resolve these issues, we need either to change the spec or the
> implementations (or both).
>
> Some observations.
>
> 1. The change only effects xs:anyURI options where you’re using a
>    relative URI and relying on it remaining relative. Probably not a high
> proportion of actual pipelines.
>
> 2. The change is silently backwards incompatible in a fairly dramatic
>    way. If you have a pipeline that falls into the category described in
> point 1, it will suddenly receive completely different URIs and may behave
> badly. It will not be the easiest bug to find, but also probably not the
> hardest.
>
> 3. There’s no way to change x:mystep so that you get the correct
>    result. There’s no way for x:mystep to “see” the base URI that was in
> effect on the (possibly implicit) p:with-option instruction that defined
> $file.
>
> 4. If we don’t change the implementations, you can get the correct
>    result by resolving the URI at the point of the call:
> file="{resolve-uri('pipe.xpl')}" But that’s a bit ugly and it’s annoying to
> have to remember to do that everytime.
>
> 5. If we do change the implementations, you can get the current result
>    by changing the type from xs:anyURI to xs:string. (That may be a little
> unsatisfying, but think of it this way: if you’re not resolving them as the
> specification requires, then they aren’t URIs from the point of view of the
> XProc.)
>
> 6. The current declaration for p:namespace-rename is in contradiction
>    with what the spec says. The $from and $to options are declared as
> xs:anyURI, so their values should be made absolute. Despite the fact that
> they’re described as URIs, in reality, almost all XML software treats
> namespace URIs as opaque strings. The namespaces recommendation
> countenances this behavior with the observation that “URIs are treated as
> strings, and they are identical if and only if the strings are identical.”
>
> 7. If we don’t change the implementations, we have to explain why
>
>    <p:store>
>      <p:with-option xml:base="/tmp/x/" name="href" select="'pipe.xpl'"/>
>    </p:store>
>
>    and
>
>    <x:mystep>
>      <p:with-option xml:base="/tmp/x/" name="file" select="'pipe.xpl'"/>
>    </p:store>
>
>    behave in completely different ways. You see, *built in* atomic steps
> *do* resolve their xs:anyURI options against the base URI as the spec
> describes, it’s only user-defined steps that don’t! (Built in atomic steps
> *can* “see” the base URI that was in effect when the option was defined.)
>
> I have vacillated mightily on this issue. Sometimes I feel like the spec
> is clear, the implementations are wrong, we must fix them. Sometimes I feel
> like the change is backwards incompatible in a really ugly way and
> backwards incompatible changes must be avoided if at all possible.
>
> Point 7 broke me. I don’t see how we can have any kind of coherent story
> with that flaw in it. I think we have to change how the implementations
> work so that p:store and x:mystep work in the same way.
>
> Counterpoint: Gerrit observed[1] in the XML Calabash issue about this
> problem, that we *could* amend the specification so that it simply says
> “user defined steps don’t work this way.” Indeed, we could, but that seems
> very unsatisfactory to me. And it wouldn’t apply to the p:variable case
> anyway.
>
> At one point in the previous thread[2] about this issue, I reported that
> the change caused no tests in the test suite to fail. Then later, I said
> “oh, I did that wrong about 30 tests fail”. I wish I’d said more about
> which ones and why because I can’t reproduce that result. I wrote two new
> tests that explicitly depend on the behavior in question and they pass or
> fail depending on how I configure my implementation, all of the other tests
> pass either way. (But what about the p:namespace-rename tests? Well, as it
> happens, my implementation of that step already treats “to” and “from” as
> strings, not URIs. You just couldn’t tell before. So they still pass either
> way.)
>
> In brief (if you’ve read this far, you may not think I’m being especially
> brief!) I’ve moved on from “is this a bug, should we fix this?” to “yes,
> this is a bug, how should we fix this?”
>
> First, the declaration of p:namespace-rename has to be changed, and I
> think the actual prose in the spec about resolving URIs could do with a
> little wordsmithing to make it clearer.
>
> Second, we have to change how the implementations work.
>
> Today, I have an option that I suspect no one but me has ever used, that
> allows you to turn on the conformant behavior. One option is simply to
> change the default behavior and provide an option to turn on the old
> behavior. I believe Achim could do something similar.
>
> That’s the easiest thing to do: we say, the implementations were wrong,
> we’ve fixed them, in the unlikely event that you need the old behavior,
> here’s an implementation-defined way to get it back.
>
> That’s possibly a little bit user-hostile. If you need the old behavior,
> you have to special case things for the processor that you’re using and if
> you switch processors you get an error, or in the worst case, just the
> wrong behavior.
>
> I make the following observation with tremendous reluctance. Given that we
> have to change the declaration of p:namespace-rename, we could publish an
> XProc 3.2 specification and say that this behavior is dependent on the
> pipeline version. Version 3.0 and 3.1 pipelines retain the old behavior,
> version 3.2 and above pipelines get the new behavior.
>
> It’s awful because the behavior is just as wrong for 3.0 and 3.1 pipelines
> as it is for 3.2 pipelines. We’ll be forcing any potential new implementors
> to copy current wrong behavior into their implementations *and* the correct
> behavior.
>
> (In principle, we could have a new flag rather than a new version, but
> then we’d have to say in the spec that if the flag was set to “old
> behavior” then that was the correct behavior and then I’d have to argue
> that in “old behavior” mode, built in steps like p:store should *also* use
> the old behavior, and…no. Just, no.)
>
> What should we do?
>
>                                         Be seeing you,
>                                           norm
>
> [1]
> https://codeberg.org/xmlcalabash/xmlcalabash3/issues/105#issuecomment-5607491
> [2] https://lists.w3.org/Archives/Public/xproc-dev/2025Jun/0007.html
>
> --
> Norm Tovey-Walsh <ndw@nwalsh.com>
> https://norm.tovey-walsh.com/
>
> > There is no monument dedicated to the memory of a committee.--Lester J.
> Pourciau
>
>

-- 
...Wendell Piez... ...wendellpiez.com...
...pellucidliterature.org... ...pausepress.org... ...github.com/wendellpiez.
..