Re: Tests recently changed with "Hidden" URL assumption ...

"David A. Lee" <dlee@calldei.com> writes:
> Suggestion:
> Put it back the way it was.
>
> Prior:
> Tests were independant of where they were run.  Atleast this test, I
> havent checked them all.
> But with this set of tests they worked and were deterministic
> reguardless of where they were run.
> The test body itself was explicit and the results were correct without
> assuming anything.

I don't think that was the case. In fact, I think a "pass" required
that your test setup be the same as mine.

The problem lines in what to do about the "correct" results against
which the pipeline outputs are compared.

I see three options:

1. <doc xml:base='http://tests.xproc.org/tests/doc/xml-base-test.xml'>

This works only if you run the test from the test website.

2. <doc xml:base='file:///path/to/tests/doc/xml-base-test.xml'>

This works only if you've installed the tests in /path/to on your
local machine. (And you're not on a Windows box where drive letters
will come into play.)

3. <doc xml:base='doc/xml-base-test.xml'>

This works only if we add a rule that says that xml:base attributes in
test suite results are to be made absolute before comparison. We could
do that, but it's not a standard requirement so it would be extra work
for implementors building a test harness.

Of the three choices, I think option 1 represents the best compromise
between practicality and ease of use.

> Tests (atleast this one) are non-deterministic.  They produce
> different results depending on where they were run.  In order to
> "pass" tests MUST be run with the base URI set to the http://tests
> site
> reguardelss of where they were run.  Tests now have an undocumented
> hidden assumption which is completely unnecessary and adds
> considerible complexity and confusion.
> IMHO, the change was not an improvement, it was a regression.

I think it was an improvement because it moved the non-determinism
From the layout of directories on my laptop to the standard test suite
location :-)

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com> | The First Amendment is often
http://nwalsh.com/            | inconvenient. But that is besides the
                              | point. Inconvenience does not absolve
                              | the government of its obligation to
                              | tolerate speech.--Justice Anthony
                              | Kennedy, in 91-155