RE: Submitting test suite results to tests.xproc.org

> Dear implementors,
> 
> I've cooked up an XML format for reporting test suite 
> results. It's defined
> by the schema at 
> http://tests.xproc.org/schemas/testreport.rnc (or .rng).
> (Alas, there's no documentation yet.)
> 

Great news! But the schemas still don't seem to be there (nor .rnc nor
.rng)...

> For an example of the format, you can download the XML Calabash report
> From http://tests.xproc.org/results/calabash/report.xml

I don't think it's the best example of the format, it contains too few
failures :)

> I'm open to suggestions for changes/improvements.

I think that for the tests that are expected to fail, the schema should
distinguish between three possible cases:

- processor did not fail
- pricessor did fail with proper error code
- processor failed, but with a different error code (and interpret this
as 50% success or something like that?)

(But perhaps the schema already takes this into account, I can't check.)

> 
> As an added incentive to use the format :-), I've created a web
> service that will allow you to submit results in that format to
> tests.xproc.org and have the results posted immediately. Like
> the XML Calabash report at http://tests.xproc.org/results/calabash/
> 
> Send me the short name that you would like to use to identify your
> implementation and a password that you'd like and I'll set it up.
> 
> The short name will appear in the URI:
> 
>   http://tests.xproc.org/results/<shortname>

I will submit the results as soon as I implement support for the schema.

> And finally, here's the pipeline that I use to run the tests and
> submit the results. It's actually a fairly interesting pipeline.
> Questions and/or suggestions for improvements most welcome.
> 
> <p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
> 		xmlns:c="http://www.w3.org/ns/xproc-step"
> 		xmlns:cx="http://xmlcalabash.com/ns/extensions"
> 		exclude-inline-prefixes="c cx"
> 		name="main">
...
> </p:declare-step>

Indeed interesting. Maybe it is not a bad idea to post this pipeline on
tests.xproc.org as the preferred way of submitting the results :)


Once again, thanks for the effort.

Regards,
Vojtech

Received on Thursday, 4 December 2008 13:15:40 UTC