Re: PipX, a portable library of XProc pipelines and steps

Thanks for the feedback.

Yes, I agree it is rather verbose. I think it makes it quite flexible, but
it would probably be nice to have some syntactic sugar. For instance;

<x:error code="pipx:no-parameter"/>

could be syntactic sugar for

<x:expect type="xpath"
test="count(/c:errors/c:error[@code='pipx:no-parameter']) &gt; 0"/>


I think it is useful to be able to define a context, rather than referring
directly to output ports etc. from the assertion element. You often want to
perform multiple assertions against the same context, so instead of
referring to the port (or whatever) on each assertion, it's convenient to
specify the context first. In addition to ports and errors, you may want to
check the directory structure (either flat or recursive), and file contents
(xml, text, binary) that are stored to disk by the xproc script. For
instance, you may want to check that all images referenced from a document
exists on disk.

It's not perfect, but it works. It would be cool to see other
implementations of xproc testing tools though :).


Jostein


On 21 February 2014 13:53, Florent Georges <fgeorges@fgeorges.org> wrote:

>   Thanks, Jostein.  I gave it a try, and put the resulting HTML
> report at http://pipx.org/tmp/pipx-parameter.html.  This is what
> I get if I execute the following command:
>
>     calabash \
>       -o html=pipx-parameter.html \
>       -o result=pipx-parameter-report.xml \
>       -o junit=pipx-parameter-junit.xml \
>       -i pipx-parameter.xprocspec \
>       $(XPROCSPEC)
>
>   But I have serious concerns about the language itself.  I find
> it rather verbose, non composable, and it lacks assertions as
> crutial as on errors, apparently.  I suffers from the same flaws
> as XSpec does (I will not blame you for those, surely :-p).
>
>   If we look at an example, the test in XProcSpec looks like:
>
>     <x:scenario label="with a required parameter, none provided.">
>        <!--  <p>Requires a parameter <code>foo</code> but does not
> provide any parameter. The error <code>pipx:no-parameter</code> must
> be thrown.</p> -->
>        <x:call step="pipx:parameter">
>           <x:option name="param-name" select="'foo'"/>
>           <x:option name="required" select="'true'"/>
>           <!-- it appears to be a bug in xprocspec with assigning
> documents to the primary parameter port, this is how it should've
> worked: -->
>           <!-- <x:input port="parameters"/> -->
>           <!-- assigning a parameter with an unexpected name works for
> now: -->
>           <x:param name="other" select="''"/>
>        </x:call>
>        <x:context label="the error document">
>           <x:document type="errors"/>
>        </x:context>
>        <x:expect label="should contain the error 'pipx:no-parameter'"
> type="xpath" test="count(/c:errors/c:error[@code='pipx:no-parameter'])
> &gt; 0"/>
>     </x:scenario>
>
> and the one I wrote:
>
>     <t:test code="param-003" step="pipx:parameter">
>        <t:title>Required parameter, none provided.</t:title>
>        <t:documentation>
>           <p>Requires a parameter foo but does not provide any parameter.
>              The error <code>pipx:no-parameter</code> must be thrown.</p>
>        </t:documentation>
>        <pipx:parameter param-name="foo" required="true">
>           <p:input port="parameters">
>              <p:empty/>
>           </p:input>
>        </pipx:parameter>
>        <t:error code="pipx:no-parameter"/>
>     </t:test>
>
>   To be fair, if we get rid of all documentation and labels, it
> becomes the following in XProcSpec:
>
>     <x:scenario>
>        <x:call step="pipx:parameter">
>           <x:option name="param-name" select="'foo'"/>
>           <x:option name="required" select="'true'"/>
>           <x:param name="other" select="''"/>
>        </x:call>
>        <x:context>
>           <x:document type="errors"/>
>        </x:context>
>        <x:expect type="xpath"
> test="count(/c:errors/c:error[@code='pipx:no-parameter']) &gt; 0"/>
>     </x:scenario>
>
> and the one I wrote:
>
>     <t:test>
>        <pipx:parameter param-name="foo" required="true">
>           <p:input port="parameters">
>              <p:empty/>
>           </p:input>
>        </pipx:parameter>
>        <t:error code="pipx:no-parameter"/>
>     </t:test>
>
>   I think the last one is more flexible, as it allows any sub-
> pipeline to be tested (which makes the test more clear as well).
> And the error assertion is definitely more clear (and I would say
> error prone in XProcSpec).
>
>   The x:context element is really bizarre.  It used to set the
> context for a test of a transformation in XSpec.  Here it seems
> to set the "context" for the assertions?
>
>   I like the JUnit output, for a general purpose unit test
> framework but I don't think it is useful in this case.  I like
> the hierarchical report as well.
>
>   So even though I would rather call this kind of solution an
> issue, I think in this case it makes sense to keep this question
> open, having both test suites set up (this is already the case),
> and see what people say in the near, near, near future.  So we
> can make a choice based on (little) experience.
>
>   Regards,
>
> --
> Florent Georges
> http://fgeorges.org/
> http://h2oconsulting.be/
>
>
> On 20 February 2014 19:39, Jostein Austvik Jacobsen wrote:
> > - Download xprocspec (or clone its repository):
> > http://daisy-consortium.github.io/xprocspec/
> > - If you want validation of the xprocspec documents (useful when
> authoring
> > them - gives autocomplete etc); add xprocspecs catalog.xml to your list
> of
> > xml catalogs (in oxygen; edit "~/Oxygen XML Editor
> > 15/frameworks/catalog.xml" and add "<nextCatalog
> >
> catalog="/home/USERNAME/xprocspec/xprocspec/src/main/resources/META-INF/catalog.xml"/>"
> > to the end.
> > - To execute a xprocspec test, run
> >
> "~/xprocspec/xprocspec/src/main/resources/content/xml/xproc/xprocspec.xpl"
> > with your xprocspec test document on the primary input source port. If
> > you're on Windows you should set the optional "temp-dir" option to
> something
> > more appropriate than the default "file:/tmp/"
> > - xprocspec has three secondary output ports; result, html and junit. The
> > "result" port contains the raw output from the xprocspec evaluation. It's
> > probably not useful to most users. The "junit" is mainly intended for
> maven
> > integration but can be useful if you need a machine-readable format. The
> > "html" port is probably what you want though; it contains a single html
> > document containing the test report.
> >
> > I've set up a scenario in oXygen called "Basic xprocspec test" where I
> use
> > the current document as input and point the output ports to files in
> > file:/tmp/. So whenever I have a xprocspec test document open in oXygen I
> > can run that scenario, then go to my browser and refresh
> > "file:///tmp/xprocspec-html.html" to see the results. You could also
> > instruct oXygen to automatically open the html file I suppose.
> >
> > I also have a oXygen scenario that lets me run the test while viewing the
> > XProc document itself (to avoid switching between tabs when repeatedly
> > running tests), but that's based on file nameing conventions etc. so I
> won't
> > bother you with the details (unless you're interested).
> >
> > Hope this helps.
> >
> >
> >
> > wrt verbosity: I'm not sure how to make it less verbose without removing
> > functionality or overloading elements with attributes. But I'm open to
> > suggestions!
> > wrt subpipelines: If you want to test subpipelines, then you can create a
> > small wrapper script and test its inputs and outputs (which you might
> want
> > to expose in your library anyway if it's a common pattern). By the way,
> > xprocspec supports custom implementations of assetions if that helps.
> >
> >
> > Jostein
> >
> >
> > On 20 February 2014 19:10, Florent Georges <fgeorges@fgeorges.org>
> wrote:
> >>
> >> On 20 February 2014 18:46, Jostein Austvik Jacobsen wrote:
> >>
> >> > Here's an equivalent xprocspec test:
> >> > https://github.com/fgeorges/pipx/pull/1/files
> >>
> >>   Thanks.  I accepted your pull request, I'll have a look later.  Can
> >> you explain how to run it?  At first glance, I would say it is a bit
> >> verbose, and it has the same drawback (someone would say advantage) as
> >> XSpec has: it allows you only to execute one step, not a subpipeline.
> >> What if you want to test for instance:
> >>
> >>     <pipx:step-a .../>
> >>     <pipx:step-b .../>
> >>
> >> instead of just calling either pipx:step-a or pipx:step-b?
> >>
> >>   I will try to update http://pipx.org/progress.html soon also, but
> >> for now, I have to call it a day...
> >>
> >>   Regards,
> >>
> >> --
> >> Florent Georges
> >> http://fgeorges.org/
> >> http://h2oconsulting.be/
> >
> >
>