Re: Reporting on QT3 test results for the 3.1 Specs

As discussed this morning, I think it would also be useful if we could eliminate entirely from the report those tests that are not applicable to XP31 or XQ31 processors. It's right that we have kept XQ10/30  or XP20/30 versions of tests in the test suite where the expected results are different, but we shouldn't be counting these as tests which no-one has passed.

Michael Kay
Saxonica 

> On 18 Jan 2016, at 14:40, O'Neil Delpratt <oneil@saxonica.com> wrote:
> 
> Dear All,
> 
> I have re-generated the reports on the implementation results received for the QT3 test suite. See update: http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html <http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html>
> We now have three implementations for XQuery 3.1, with one being anonymous. However we only have one for XPath 3.1.
> 
> We have been working on the main page today. Specifically the stats on the optional features. We found that there were some overlap reported in the passes and failures. For example under ‘staticTyping’ it is reported that there are 75 test not being passed by any implementation. Under the 'higherOrderFunction’ feature we have 18 test passed by no implementation. If you inspect the higherOrderFunction tests more closely we will see that 17 of these tests depend on staticTyping therefore the underlying reason for the higherOrderFunction failures is to do with staticTyping. With what I have said in mind we think it helpful to show the combination of optional features for the test case.
> 
> I welcome feedback on your preference of presentation on the Optional Features section. I find 2) much more useful:
> 
> 1) http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optFeatures <http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optFeatures>
> 2) http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optComboFeatures <http://dev.w3.org/2011/QT3-test-suite/ReportingResults31/report.html#optComboFeatures>
> 
> 
> Using 1) or 2) we observe that we have 75 tests with dependency on staticTyping not being passed by any implementation (including those with combinations of other dependencies).  I would like to ask if it still merits having tests on static typing?
> 
> kind regards,
> 
> 
> -------------------------------
> O'Neil Delpratt
> Software Developer, Saxonica Limited 
> Email: oneil@saxonica.com <mailto:oneil@saxonica.com>
> Twitter: https://twitter.com/ond1 <https://twitter.com/ond1>
> Tel: +44 118 946 5894
> Web: http://www.saxonica.com <http://www.saxonica.com/>
> Saxonica Community site: http://dev.saxonica.com <http://dev.saxonica.com/>
> Bug tracking site: https://saxonica.plan.io/ <https://saxonica.plan.io/>
> 
> 
>