more on xproc unit tests

Hello All,

I decided to get some more real world parallel experience in unit
testing declarative markup...it seemed to make sense to me to
investigate other unit testing approaches before continue proposing a
direction for XProc.

The one that stood out as slightly comparable (as in it being used in
anger and is part of a project that obliquely can process XML in a
somewhat linear fashion)...is Ant.

Ant comes now with a useful unit testing suite called antunit
(http://ant.apache.org/antlibs/antunit/index.html).

As the basis for my example, this week I set about writing unit tests
in antunit, for eXist xmldb Ant tasks....this , in itself, is
admittedly removed from XProc...but it gave me the 'flavour' of what
unit testing in XProc might start feeling like.

I didn't try to replicate xml process flows in ant..just wanted to
test Ant tasks which could be considered 'step's in the XProc sense.

if interested, I have submitted these antunit tests to eXist xmldb here
http://sourceforge.net/tracker/index.php?func=detail&aid=1743796&group_id=17691&atid=317691

now for some small (perhaps obvious) conclusions;

* In Ant unit testing, with antuint, we have 'access' to make
assertions on most of Ant's datatypes and it still 'felt' like
functional testing...in XProc we really only have access to output
from pipelines....so I can imagine that testing would even be more
functional in flavour.

* In XProc there would be no mechanism to directly make assertions on
things like options, parameters, or directly examining inputs;
everything would essentially be a comparison between 2 XML
documents...which practically means using <p:equal/>

* <p:equal/> gets around the XML only output rule for XProc...by using
a c:result is that right?...also that reminds me, what is the output
of steps like <p:validate-relax-ng/> and <p:validate-xml-schema/>
steps...same sort of thing?

* how would we write assertions that doesn't interfere with a
pipelines flow (that does actual work)...I wouldn't want to sandwich a
step in between two steps...I simply would like to have a branch for
the assertion

ok.....lets move onto how different approaches would work and look
like in real world examples.

here is a basic example taken from the current XProc spec;

<p:pipeline name="pipeline" xmlns:p="http://www.w3.org/2007/03/xproc">
<p:input port="document"/>
<p:input port="stylesheet"/>

<p:input port="expectedresult1"/>
<p:input port="expectedresult2"/>
<p:input port="expectedresult3"/>

<p:output port="result"/>

<p:xinclude>
  <p:input port="source">
    <p:pipe step="pipeline" port="document"/>
  </p:input>
</p:xinclude>

<p:equal>
     <p:input port="source"/>
     <p:input port="expectedresult1"/>
     <p:output port="result"/>
     <p:option name="fail-if-not-equal" value="no"/>
</p:equal>

<p:validate-xml-schema>
  <p:input port="schema">
    <p:document href="http://example.com/path/to/schema.xsd"/>
  </p:input>
</p:validate-xml-schema>

<p:equal>
     <p:input port="source"/>
     <p:input port="expectedresult2"/>
     <p:output port="result"/>
     <p:option name="fail-if-not-equal" value="no"/>
</p:equal>

<p:xslt>
  <p:input port="stylesheet">
    <p:pipe step="pipeline" port="stylesheet"/>
  </p:input>
</p:xslt>

<p:equal>
     <p:input port="source"/>
     <p:input port="expectedresult3"/>
     <p:output port="result"/>
     <p:option name="fail-if-not-equal" value="no"/>
</p:equal>

</p:pipeline>

So the main question here (other then why anyone would want to do this
pedantic style of testing)

is ...will such tests interfere with pipeline process flow of the
actual XML document we want to work with (not the c:result document)?

we could opt for a separate p:assert step that would not forward its
output to next step by default, effectively being a 1 step branch off
the main pipeline flow (btw is this possible....e.g. have a branching
pipeline process along with main pipeline trunk?)

<p:pipeline name="pipeline" xmlns:p="http://www.w3.org/2007/03/xproc">
<p:input port="document"/>
<p:input port="stylesheet"/>

<p:input port="expectedresult1"/>
<p:input port="expectedresult2"/>
<p:input port="expectedresult3"/>

<p:output port="result"/>

<p:xinclude>
  <p:input port="source">
    <p:pipe step="pipeline" port="document"/>
  </p:input>
</p:xinclude>

<p:assert message="test1 failed">
     <p:input port="source"/>
     <p:input port="expectedresult1"/>
     <p:output port="result"/>
     <p:option name="fail-if-not-equal" value="no"/>
</p:assert>

<p:validate-xml-schema>
  <p:input port="schema">
    <p:document href="http://example.com/path/to/schema.xsd"/>
  </p:input>
</p:validate-xml-schema>

<p:assert  message="test2 failed">
     <p:input port="source"/>
     <p:input port="expectedresult2"/>
     <p:output port="result"/>
     <p:option name="fail-if-not-equal" value="no"/>
</p:assert>

<p:xslt>
  <p:input port="stylesheet">
    <p:pipe step="pipeline" port="stylesheet"/>
  </p:input>
</p:xslt>

<p:assert  message="test3 failed">
     <p:input port="source"/>
     <p:input port="expectedresult3"/>
     <p:output port="result"/>
     <p:option name="fail-if-not-equal" value="no"/>
</p:assert>

</p:pipeline>

still pedantic but makes life easier if the p:asserts do not
participate in flowing its output to next steps input....unless of
course we would want that....which probably means there is a p:option
needed.

I also added a @message attribute because I would want something like
that in a <p:assert/>

This style of assertion still feels a bit clumsy and ......too chunky
for all scenarios.

For example, I may want to test some finer grain then a whole XML
document versus another

    * that the xslt stylesheet has right version number in stylesheet port

    * that the document bound to the source port has correct XInclude
namespace defined

    * that there exists  certain number of XML elements

    * that an xml element has a certain value

    * be able to compare 2 values (not xml)

in fact a lot of the type of tests we would like to do (after clumsy
equivalence) is schematron style assertions.

This seems to jig with the 'functional' feeling of the antunit tests as well.

This is not so hard, and I think that this is quite doable by taking
advantage of @select on input.

Now I have left out the discussion testing basic elements of XProc
itself, because I think directly testing inputs and outputs, options,
parameters, and other things are probably a bit too much to ask for at
this stage of XProc.

Conclusion:

A minimal proposal would be the capability for p:equal to not fully
participate in pipeline processing, e.g. an option that basically says
evaluate and report and pass the previous output along....better yet I
would propose a separate p:assert, but if a p:option existed on
p:equal then we get most scenarios needed for unit testing. In
addition a @message attribute would be useful as well.

I did base my thinking on previous XProc spec, as I see we have a new
somewhat different version which I will need to get my head around
this week.

I am also now starting to build an XProc test suite. Using the
existing W3 defined one as as starting point.

cheers, Jim Fuller

Received on Tuesday, 26 June 2007 20:43:33 UTC