Re: Reporting on QT3 test results for the 3.1 Specs

Another area where it would be nice to improve the reporting is for "alternative feature" tests. For example, if we have versions of a test for Unicode 5.2 and Unicode 6.0, and it's implementation-defined which version of Unicode a processor must support, then we don't want any red ink on the report just because no-one has chosen to support Unicode 5.2.

It's tricky to capture the metadata for this, we don't have any obvious way to say "a product must pass test XYZ-0123a or test XYZ-0123b, and for test reporting purposes, passing either of these counts as success".

Similarly, I don't think it's relevant to report how many products support Thai numbering (for example). We're listing this in the test report under a heading "Optional Features", but it's not an Optional Feature in the sense of 

https://www.w3.org/TR/2015/CR-xquery-31-20151217/#id-conform-optional-features

I would venture to suggest that for reporting purposes, we exclude all tests that have dependencies on things that are "implementation-defined behaviour" as distinct from "optional features" - using the latter term in the sense it is used in the spec. Or at any rate, put the "implementation-defined" aspects in a different place, and don't use red ink where no products are passing the test, because it's not a success criterion that they should.

Michael Kay
Saxonica


> On 18 Jan 2016, at 14:40, O'Neil Delpratt <oneil@saxonica.com> wrote:
> 
> Dear All,
> 
> I have re-generated the reports on the implementation results received for the QT3 test suite. See update: http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html <http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html>
> We now have three implementations for XQuery 3.1, with one being anonymous. However we only have one for XPath 3.1.
> 
> We have been working on the main page today. Specifically the stats on the optional features. We found that there were some overlap reported in the passes and failures. For example under ‘staticTyping’ it is reported that there are 75 test not being passed by any implementation. Under the 'higherOrderFunction’ feature we have 18 test passed by no implementation. If you inspect the higherOrderFunction tests more closely we will see that 17 of these tests depend on staticTyping therefore the underlying reason for the higherOrderFunction failures is to do with staticTyping. With what I have said in mind we think it helpful to show the combination of optional features for the test case.
> 
> I welcome feedback on your preference of presentation on the Optional Features section. I find 2) much more useful:
> 
> 1) http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optFeatures <http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optFeatures>
> 2) http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optComboFeatures <http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optComboFeatures>
> 
> 
> Using 1) or 2) we observe that we have 75 tests with dependency on staticTyping not being passed by any implementation (including those with combinations of other dependencies).  I would like to ask if it still merits having tests on static typing?
> 
> kind regards,
> 
> 
> -------------------------------
> O'Neil Delpratt
> Software Developer, Saxonica Limited 
> Email: oneil@saxonica.com <mailto:oneil@saxonica.com>
> Twitter: https://twitter.com/ond1 <https://twitter.com/ond1>
> Tel: +44 118 946 5894
> Web: http://www.saxonica.com <http://www.saxonica.com/>
> Saxonica Community site: http://dev.saxonica.com <http://dev.saxonica.com/>
> Bug tracking site: https://saxonica.plan.io/ <https://saxonica.plan.io/>
> 
> 
>