Re: [Zorba-coders] Re: Zorba revision 8827 XQuery/XQueryX Update Test Suite Results

The summary column reports the number of implementations that pass 100% of 
the test cases, out of the number of implementations reporting. I believe 
the column is green if we have 2 or more such implementations.

The stylesheet generating the report was created before we had test cases 
that depended on specific features, and so is not correct for the optional 
features. This may be fixed in the future, but not right now.

Feel free to report this via bugzilla if you like.

                                        -- Andrew

--------------------
Andrew Eisenberg
IBM
5 Technology Park Drive
Westford, MA  01886

andrew.eisenberg@us.ibm.com
Phone: 978-399-5158





From:
Matthias Brantner <matthias.brantner@28msec.com>
To:
Andrew Eisenberg/Concord/IBM@IBMUS
Cc:
Marian Sorin Nasoi <spungi@gmail.com>, www-archive@w3.org, zorba-coders 
<zorba-coders@lists.inf.ethz.ch>
Date:
09/17/2010 02:37 AM
Subject:
Re: [Zorba-coders] Re: Zorba revision 8827 XQuery/XQueryX Update Test 
Suite Results
Sent by:
Matthias Brantner <mbrantner@gmail.com>



Thanks Andrew. This looks good now. ;-)

BTW: what do the numbers in the "Summary" column mean? For example,
for the Revalidation feature it's red and shows "0/5".

Best

Matthias

Chief Architect

28msec Inc.
http://www.28msec.com
http://twitter.com/28msec

On Sep 16, 2010, at 11:51 PM, Andrew Eisenberg wrote:

> 
> I've updated the reports ... the red has disappeared. Thanks. 
> 
>                                        -- Andrew
> 
> --------------------
> Andrew Eisenberg
> IBM
> 5 Technology Park Drive
> Westford, MA  01886
> 
> andrew.eisenberg@us.ibm.com
> Phone: 978-399-5158
> 
> 
> 
> 
> From:          Marian Sorin Nasoi <spungi@gmail.com>
> To:            Andrew Eisenberg/Concord/IBM@IBMUS
> Cc:            daniela florescu <dflorescu@mac.com>, www-archive@w3.org, 
zorba-coders <zorba-coders@lists.inf.ethz.ch>
> Date:          09/16/2010 03:32 AM
> Subject:               Re: [Zorba-coders] Re: Zorba revision 8827 
XQuery/XQueryX Update Test Suite Results
> 
> 
> 
> 
> Hi Andrew,
> 
> sorry for the misunderstanding:
> I have regenerated the reports for XQuery Update and XQueryX Update
> test suite for Zorba revision 8827 (http://www.zorba-xquery.com/).
> 
> The results are based on the XQUTS taken from the W3C CVS on September 
13, 2010.
> 
> Thanks for the help,
> Sorin
> 
> On 9/15/10, daniela florescu <dflorescu@mac.com> wrote:
> > Sorin,
> >
> > please regenerate the results with "NOT APPLICABLE" instead of fail.
> > Please resend to Andrew after that.
> >
> > Andrew, I guess this wasn't clear enough for us.
> >
> >
> > Thanks
> > Dana
> >
> >
> > On Sep 15, 2010, at 1:38 PM, Andrew Eisenberg wrote:
> >
> >>
> >> > Hi Andrew,
> >> >
> >> > again thanks for your efforts.
> >> >
> >> > I looked over the reports and I would have some questions if I may:
> >> >
> >> > - for 'Minimal Conformance' Saxon EE is shown as passing all the
> >> > tests: how is that since for test 'namespace-errors-q16' there is 
an
> >> > opened bug #10432 ?
> >> > Zorba reports the alternate result as proposed by Markos and agreed
> >> > upon by Michael Kay but since that alternate result is not in
> >> XQUTS at
> >> > the moment, Zorba reports 'namespace-errors-q16' as failing, which
> >> is
> >> > the correct thing, right?
> >> > Am I missing something here?
> >>
> >> This too shall pass. I understand your confusion here. We published
> >> XQUTS 1.0.1, began collecting results and publishing reports,
> >> accepted bug reports, and modified the test suite. Saxon's results
> >> were submitted earlier, yours were submitted later. I'm planning to
> >> wait until Bug #10432 is resolved and then ask all vendors to
> >> resubmit their results.
> >>
> >> > - for the optional feature 'Update Facility Static Typing Feature'
> >> > Zorba mentions that it does not support this feature (see
> >> > http://dev.w3.org/2007/xquery-update-10-test-suite/results/
> >> > XQUTSReport.html#features)
> >> > yet in the first table of the report Zorba is shown in red (meaning
> >> > failed) instead of white (which stands for untested).
> >> > Could it be that there is an error in the report generation
> >> process here?
> >> > I mean shouldn't the tests for a feature that an implementation
> >> > mentions it does not support, be shown in white ?
> >>
> >> The stylesheet that generates the reports uses the pass/fail results
> >> that you provide, independent of whether the feature is supported.
> >> You could provide a result of  "not applicable" or "not tested"
> >> instead of "fail", Only "fail" generates the red shading.  You could
> >> add a comment attribute explaining that the feature is not supported
> >> if you wish.
> >>
> >> > - for the optional feature 'fn:put applied to nodes other than
> >> > document and element nodes'  there are 4 tests that Zorba 'fails'
> >> but
> >> > these tests should be run only if the XQuery processor does NOT
> >> > support the feature.
> >> > Since the tests were created by you, can you please tell us what is
> >> > the correct way to handle these tests so that we can correctly show
> >> > them as passing?
> >>
> >> As I said above, you could report a result of  "not applicable" or
> >> "not tested". You could not generate results for those tests. This
> >> looks like what the other products are doing.
> >> _______________________________________________
> >> Zorba-coders mailing list
> >> Zorba-coders@lists.inf.ethz.ch
> >> https://lists.inf.ethz.ch/mailman/listinfo/zorba-coders
> >
> >
> [attachment "Zorba_r8827_XQuery(X)_update_results_new.zip" deleted by 
Andrew Eisenberg/Concord/IBM] 
> 
> _______________________________________________
> Zorba-coders mailing list
> Zorba-coders@lists.inf.ethz.ch
> https://lists.inf.ethz.ch/mailman/listinfo/zorba-coders

Received on Friday, 17 September 2010 15:49:41 UTC