The xs:anyURI problem…

[ This turned out to be a longer message than I expected, even when I expected it to be long. Sorry about that. ]

Hello and happy seasonal greetings!

It’s been six months and we haven’t resolved this issue. It feels like we should grasp the nettle and resolve it.

To recap:

Given a step, x:mystep, declared in /path/to/lib/mystep.xpl

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:xs="http://www.w3.org/2001/XMLSchema" 
                xmlns:x="http://example.com/ns/steps"
                name="main" version="3.1" type="x:mystep"
                exclude-inline-prefixes="#all">
  <p:output port="result"/>
  <p:option name="file" as="xs:anyURI"/>
  <p:identity>
    <p:with-input><doc>{$file}</doc></p:with-input>
  </p:identity>
</p:declare-step>

And a pipeline in /path/to/pipeline.xpl:

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:xs="http://www.w3.org/2001/XMLSchema" 
                xmlns:x="http://example.com/ns/steps"
                name="main" version="3.1"
                exclude-inline-prefixes="#all">
  <p:import href="lib/mystep.xpl"/>
  <p:output port="result"/>

  <x:mystep file="pipe.xpl"/> 
</p:declare-step>

What should the output be? The answer hinges on how the xs:anyURI option “file” is handled.

Today, in my implementation and Achim’s, the output is <doc>pipe.xpl</doc>.

George Bina made the argument that this is wrong. The spec says of xs:anyURI options:

   If [an] option value is specified using a syntactic shortcut, the base URI of the step element on which the shortcut attribute appears must be used [as the base URI to resolve against].

With that interpretation, the output is

  <doc>file:/path/to/pipe.xpl</doc>

George’s argument is thorough and thoroughly supported by what the spec actually says. I think he’s right.

Gerrit noted that the relevant prose in the specification goes on to say:

  In general, whenever a relative URI appears in an xs:anyURI, its base URI is the base URI of the nearest ancestor element.

That complicates things a bit, and makes it more likely that the change proposed will have unforseen consequences in existing pipelines, because it applies in more places than just options. Consider:

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:xs="http://www.w3.org/2001/XMLSchema" 
                xmlns:x="http://example.com/ns/steps"
                name="main" version="3.1"
                exclude-inline-prefixes="#all">
  <p:import href="lib/mystep.xpl"/>
  <p:output port="result"/>

  <p:variable name="filename" as="xs:anyURI"
              xml:base="/alternate/path/" select="'pipe.xpl'"/>

  <x:mystep file="{$filename}"/> 
</p:declare-step>

Here, curiously, MorganaXProc-III (1.6.8 which is what I happen to have installed at the moment) and XML Calabash give different results. Morgana gives

  <doc>pipe.xpl</doc>

which is, I admit, what I expected. But XML Calabash gives 

  <doc>file:/path/to/pipe.xpl</doc>

This happens because after resolving the AVT to “pipe.xpl”, the expected type of the option is xs:anyURI, so it’s resolved against the current base URI.

<aside>
============================================================
In fact, the result from the first pipeline, <doc>pipe.xpl</doc>, is arguably incorrect. The correct result is also <doc>file:/path/to/pipe.xpl</doc>. The constant value is not being resolved against the base URI, but I think it should be. Let’s leave that to one side for the moment. Irrespective of which answer you think is correct, “pipe.xpl” and $filename should give the same result.
============================================================
</aside>

In any case case, the general rule for resolving base URIs says that the value of $filename should be file:/alternate/path/pipe.xpl.

To resolve these issues, we need either to change the spec or the implementations (or both).

Some observations.

1. The change only effects xs:anyURI options where you’re using a    
   relative URI and relying on it remaining relative. Probably not a high proportion of actual pipelines.

2. The change is silently backwards incompatible in a fairly dramatic  
   way. If you have a pipeline that falls into the category described in point 1, it will suddenly receive completely different URIs and may behave badly. It will not be the easiest bug to find, but also probably not the hardest.

3. There’s no way to change x:mystep so that you get the correct
   result. There’s no way for x:mystep to “see” the base URI that was in effect on the (possibly implicit) p:with-option instruction that defined $file.

4. If we don’t change the implementations, you can get the correct
   result by resolving the URI at the point of the call: file="{resolve-uri('pipe.xpl')}" But that’s a bit ugly and it’s annoying to have to remember to do that everytime.

5. If we do change the implementations, you can get the current result 
   by changing the type from xs:anyURI to xs:string. (That may be a little unsatisfying, but think of it this way: if you’re not resolving them as the specification requires, then they aren’t URIs from the point of view of the XProc.)

6. The current declaration for p:namespace-rename is in contradiction
   with what the spec says. The $from and $to options are declared as xs:anyURI, so their values should be made absolute. Despite the fact that they’re described as URIs, in reality, almost all XML software treats namespace URIs as opaque strings. The namespaces recommendation countenances this behavior with the observation that “URIs are treated as strings, and they are identical if and only if the strings are identical.”

7. If we don’t change the implementations, we have to explain why

   <p:store>
     <p:with-option xml:base="/tmp/x/" name="href" select="'pipe.xpl'"/>
   </p:store>

   and 

   <x:mystep>
     <p:with-option xml:base="/tmp/x/" name="file" select="'pipe.xpl'"/>
   </p:store>

   behave in completely different ways. You see, *built in* atomic steps *do* resolve their xs:anyURI options against the base URI as the spec describes, it’s only user-defined steps that don’t! (Built in atomic steps *can* “see” the base URI that was in effect when the option was defined.)

I have vacillated mightily on this issue. Sometimes I feel like the spec is clear, the implementations are wrong, we must fix them. Sometimes I feel like the change is backwards incompatible in a really ugly way and backwards incompatible changes must be avoided if at all possible.

Point 7 broke me. I don’t see how we can have any kind of coherent story with that flaw in it. I think we have to change how the implementations work so that p:store and x:mystep work in the same way.

Counterpoint: Gerrit observed[1] in the XML Calabash issue about this problem, that we *could* amend the specification so that it simply says “user defined steps don’t work this way.” Indeed, we could, but that seems very unsatisfactory to me. And it wouldn’t apply to the p:variable case anyway.

At one point in the previous thread[2] about this issue, I reported that the change caused no tests in the test suite to fail. Then later, I said “oh, I did that wrong about 30 tests fail”. I wish I’d said more about which ones and why because I can’t reproduce that result. I wrote two new tests that explicitly depend on the behavior in question and they pass or fail depending on how I configure my implementation, all of the other tests pass either way. (But what about the p:namespace-rename tests? Well, as it happens, my implementation of that step already treats “to” and “from” as strings, not URIs. You just couldn’t tell before. So they still pass either way.)

In brief (if you’ve read this far, you may not think I’m being especially brief!) I’ve moved on from “is this a bug, should we fix this?” to “yes, this is a bug, how should we fix this?”

First, the declaration of p:namespace-rename has to be changed, and I think the actual prose in the spec about resolving URIs could do with a little wordsmithing to make it clearer.

Second, we have to change how the implementations work.

Today, I have an option that I suspect no one but me has ever used, that allows you to turn on the conformant behavior. One option is simply to change the default behavior and provide an option to turn on the old behavior. I believe Achim could do something similar.

That’s the easiest thing to do: we say, the implementations were wrong, we’ve fixed them, in the unlikely event that you need the old behavior, here’s an implementation-defined way to get it back.

That’s possibly a little bit user-hostile. If you need the old behavior, you have to special case things for the processor that you’re using and if you switch processors you get an error, or in the worst case, just the wrong behavior.

I make the following observation with tremendous reluctance. Given that we have to change the declaration of p:namespace-rename, we could publish an XProc 3.2 specification and say that this behavior is dependent on the pipeline version. Version 3.0 and 3.1 pipelines retain the old behavior, version 3.2 and above pipelines get the new behavior.

It’s awful because the behavior is just as wrong for 3.0 and 3.1 pipelines as it is for 3.2 pipelines. We’ll be forcing any potential new implementors to copy current wrong behavior into their implementations *and* the correct behavior.

(In principle, we could have a new flag rather than a new version, but then we’d have to say in the spec that if the flag was set to “old behavior” then that was the correct behavior and then I’d have to argue that in “old behavior” mode, built in steps like p:store should *also* use the old behavior, and…no. Just, no.)

What should we do?

                                        Be seeing you,
                                          norm

[1] https://codeberg.org/xmlcalabash/xmlcalabash3/issues/105#issuecomment-5607491
[2] https://lists.w3.org/Archives/Public/xproc-dev/2025Jun/0007.html

--
Norm Tovey-Walsh <ndw@nwalsh.com>
https://norm.tovey-walsh.com/

> There is no monument dedicated to the memory of a committee.--Lester J. Pourciau

Received on Wednesday, 24 December 2025 12:02:54 UTC