W3C home > Mailing lists > Public > xproc-dev@w3.org > December 2008

Re: Tests recently changed with "Hidden" URL assumption ...

From: David A. Lee <dlee@calldei.com>
Date: Tue, 30 Dec 2008 07:57:30 -0500
Message-ID: <8A7A01C1B96B4490A3DB4BDA9C28ACBF@calldei.com>
To: "Norman Walsh" <ndw@nwalsh.com>, "XProc Dev" <xproc-dev@w3.org>


Quote NW:

> I have discovered that some tests have changed (I think recently)

Not very recently.
----------
Dec 4, I guess recent is relevent.  Its just yesterday I refreshed and 
discovered it.



> I liked it the way it was before where there were no such hidden 
> requirements.
>
> Making the input URL's relative is great, but only if you can
> somehow manage comparing the outputs so they dont depend on the
> previous assumptions.

I guess I'd be open to suggestions on that point, but for the purposes
of comparison, it was expedient to simply run the tests from the
website.

For my own purposes, I sometimes run the tests locally while I'm
debugging and I just ignore the failures that I know are caused by
that.
-------------

Suggestion:
Put it back the way it was.

Prior:
Tests were independant of where they were run.  Atleast this test, I havent 
checked them all.
But with this set of tests they worked and were deterministic reguardless of 
where they were run.
The test body itself was explicit and the results were correct without 
assuming anything.

Now:
Tests (atleast this one) are non-deterministic.  They produce different 
results depending on where they were run.  In order to "pass" tests MUST be 
run with the base URI set to the http://tests site
reguardelss of where they were run.  Tests now have an undocumented hidden 
assumption which is completely unnecessary and adds considerible complexity 
and confusion.
IMHO, the change was not an improvement, it was a regression.


Compromise:
If for some overwhelming reason you think using the relative URI's in the 
tests are necessary, or better, then I suggest you put the base URI in the 
test as well so that programatically a test process
can figure out what the relative URI is relative to to produce to "correct" 
results.
Simply "ignoring failures" because you "know" its OK ... sometimes ... is 
not a good strategy ...
especially for us poor fools who are using the tests to figure out how xproc 
is supposed to work.
But I suggest this is unnecessary complexity and work when they worked fine 
as full URL's before.
Tests that are more sensitive, rather then less , to initial conditions and 
environemnt make the
implementors job very frustrating.  Especially when they used to work then 
dont, I assume its my code.   In this particular case, I assert the test is 
actually broken now until you add something to the test structure and 
documentation to make explicit the implicit assumputions which are now coded 
into the test.  Or just put it back and avoid all that work.

-----------------------------------------------------------
David A. Lee
dlee@calldei.com
http://www.calldei.com
http://www.xmlsh.org
Received on Tuesday, 30 December 2008 12:58:07 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Tuesday, 30 December 2008 12:58:08 GMT