AW: Regenerating the tests for spec examples

For those who are interested, I have written a little XQuery script which can be used to run all the tests of the "xpath-functions-40/src/function-catalog.xml" document with BaseX out of the box. It certainly runs with Saxon and other processors as well if small adjustments are made (maybe it suffices to replace "xquery:eval" with the equivalent function).

Bear with me: I bet I’ve overlooked some details of the test syntax. I’m looking forward to feedback and revised versions of the script.

Best,
Christian
________________________________________

Von: Michael Kay <mike@saxonica.com>
Gesendet: Samstag, 6. Mai 2023 21:01
An: public-xslt-40@w3.org
Betreff: Regenerating the tests for spec examples

I have been working on regenerating the tests for spec examples that are marked up with fos:test markup.

To run this requires a schema-aware transformation. Using Saxon-EE, run net.sf.saxon.Transform in directory $repo/specifications/xpath-functions-40 with options:

-t -xsl:style/generate-qt3-test-set.xsl -s:src/function-catalog.xml -val:strict -o:temp.xml

First attempt failed with a validation error on the function catalog: there is an IDREF for parse-html-options but no matching ID. I cobbled together a fix.

Next attempt failed saying:

Cannot validate <Q{.../xpath-functions}analyze-string-result>: no element declaration available

Added an xsl:import-schema for the schema for the xpath-functions namespace.

The stylesheet now runs without error, producing a new version of the test-set app-spec-examples.

Running this test set with the current build of Saxon 12 produces:

Result: 557 successes, 44 failures, 0 incorrect ErrorCode, 81 not run

I've listed the test failures below. I now need to go through them seeing which are bugs or absent features in Saxon, and which are errors in the tests. I don't propose to make any changes to the actual examples in the spec until PR475 is applied, as there's too much risk of editing conflicts.

I also need to check whether there are more examples that could be marked up in a way that allows automated testing. And to look at the reason for the "81 not run", which generally indicates a dependency that Saxon doesn't satisfy.

By the way, we've always offered a free Saxon-EE license to anyone who needs it in the course of W3C standards development, to be used for that purpose only. Let me know if you want one.

Michael Kay
Saxonica

Test failures:

  fo-test-fn-contains-sequence-012
  fo-test-fn-deep-equal-009
  fo-test-fn-deep-equal-012
  fo-test-fn-deep-equal-013
  fo-test-fn-deep-equal-014
  fo-test-fn-ends-with-sequence-008
  fo-test-fn-ends-with-sequence-011
  fo-test-fn-expanded-QName-001
  fo-test-fn-expanded-QName-002
  fo-test-fn-identity-003
  fo-test-fn-items-at-002
  fo-test-fn-iterate-while-004
  fo-test-fn-replace-009
  fo-test-fn-replace-010
  fo-test-fn-replace-011
  fo-test-fn-replace-012
  fo-test-fn-replace-013
  fo-test-fn-slice-001
  fo-test-fn-slice-002
  fo-test-fn-slice-003
  fo-test-fn-slice-004
  fo-test-fn-slice-005
  fo-test-fn-slice-006
  fo-test-fn-slice-007
  fo-test-fn-slice-008
  fo-test-fn-slice-009
  fo-test-fn-slice-010
  fo-test-fn-slice-011
  fo-test-fn-slice-012
  fo-test-fn-slice-013
  fo-test-fn-slice-014
  fo-test-fn-slice-015
  fo-test-fn-slice-016
  fo-test-fn-slice-017
  fo-test-fn-slice-018
  fo-test-fn-slice-019
  fo-test-fn-slice-020
  fo-test-fn-starts-with-sequence-008
  fo-test-fn-starts-with-sequence-011
  fo-test-map-of-001
  fo-test-map-of-002
  fo-test-map-of-003
  fo-test-map-values-001
  fo-test-map-values-002