Re: Test suite status

On Montag, 14. September 2009, Ian Horrocks wrote:
> Excellent -- thanks!
>
> Regarding the Unsupported OWL Full tests, IMHO these should not be
> given approved status. If we were to do so, we would fail to
> distinguish tests for which we have concrete evidence of
> correctness---that is, system(s) passing the tests---from those for
> which we simply don't have any evidence of incorrectness. I think
> that this is a useful distinction and should be maintained.
>
> I don't see why we don't leave the Unsupported OWL Full tests as
> "proposed". This seems to describe their status rather well --
> someone has proposed this test, and we don't yet have sufficient
> evidence to approve or reject the proposal.

Fine with me. One more OWL Full test (WebOnt-I4.6-003) is completely supported 
by Jena, so I propose to approve this test if there are no objections.

Markus

>
> Ian
>
> On 11 Sep 2009, at 21:45, Markus Krötzsch wrote:
> > Update:
> >
> > * Tests were approved as suggested and the webpage is updated.
> > * The list of the 6 DL tests that still need more support is at [1].
> > * One more test can be "auto-approved": New-Feature-
> > TopObjectProperty-001
> >
> > Now the majority (82 tests) of our proposed but unsupported tests
> > belongs to
> > OWL Full, where all but one were inherited from WebOnt.
> >
> > [1]
> > http://www.w3.org/2007/OWL/wiki/
> > Test_Suite_Status#OWL_2_DL_Proposed_Test_Cases
> >
> > On Donnerstag, 10. September 2009, Markus Krötzsch wrote:
> >> Update:
> >>
> >> * The official status is called "approved."
> >>   So substitute "accept" by "approve" in my previous mails.
> >> * New results for Pellet allow two more tests to be approved:
> >>   New-Feature-BottomObjectProperty-001
> >>   New-Feature-BottomDataProperty-001
> >>
> >> I will send another notice when the test result page has been
> >> updated with
> >> the new approved tests.
> >>
> >> Markus
> >>
> >> On Mittwoch, 9. September 2009, Markus Krötzsch wrote:
> >>> Of the remaining proposed test cases, there are roughly three kinds:
> >>>
> >>>
> >>> == Almost acceptable DL tests ==
> >>>
> >>> The following tests already have some support, but not quite
> >>> enough. I am
> >>> sure that Pellet or FaCT++ could be made to pass these:
> >>>
> >>> * New-Feature-ObjectPropertyChain-BJP-002
> >>>   this is now passed by HermiT and by REL
> >>>   (but the Exit Criteria require two DL systems, so REL does not
> >>> suffice)
> >>> * New-Feature-BottomObjectProperty-001 and
> >>> New-Feature-BottomDataProperty-001 are passed only by HermiT, but
> >>> this
> >>> does not seem to be such a hard reasoning task, after all ;-)
> >>
> >> These are now passed by Pellet, so I will change their status to
> >> approved
> >> together with the remaining set unless anybody objects.
> >>
> >> I forgot some "almost acceptable" DL tests yesterday
> >>
> >>> == Almost acceptable OWL Full tests ==
> >>>
> >>> One passing implementation -- I guess this is enough for OWL Full
> >>> tests
> >>> to be acceptable.
> >>>
> >>> * WebOnt-I4.6-003
> >>> * WebOnt-I4.6-005
> >>> * WebOnt-equivalentClass-008
> >>> * WebOnt-miscellaneous-302
> >>>
> >>>
> >>> == Unsupported OWL Full tests ==
> >>>
> >>> And then there is a large amount of WebOnt tests for which the
> >>> current
> >>> OWL Full tools are all incomplete. You can see these tests at
> >>> [1]. But
> >>> the Exit Criteria do not require tools to pass all OWL Full
> >>> tests, so we
> >>> could also move forward accepting them if nobody objects (they are
> >>> proposed and WebOnt- accepted, so they have at least seen some human
> >>> checking and there are some old implementations that have passed
> >>> some of
> >>> them).
> >>>
> >>>
> >>> Markus
> >>>
> >>> [1]
> >>> http://www.w3.org/2007/OWL/wiki/
> >>> Test_Suite_Status#OWL_2_Full_Proposed_Tes
> >>> t_ Cases
> >>>
> >>> On Mittwoch, 9. September 2009, Ian Horrocks wrote:
> >>>> Hi Markus,
> >>>>
> >>>> Thanks for taking care of this -- the test results look pretty
> >>>> impressive.
> >>>>
> >>>> As far as proposed tests are concerned, we have generally been
> >>>> accepting any test that passes the basic "eyeball" test and that is
> >>>> successfully passed by two implementations. If you can send round a
> >>>> list of proposed test that fall into this category then I think
> >>>> that
> >>>> it is reasonable to promote them to approved status unless there
> >>>> is a
> >>>> positive objection.
> >>>>
> >>>> Thanks,
> >>>> Ian
> >>>>
> >>>> On 9 Sep 2009, at 09:19, Markus Krötzsch wrote:
> >>>>> After recent updates, the test suite now is in good shape
> >>>>> regarding
> >>>>> accepted
> >>>>> tests [1]. There are still a number of proposed tests which have
> >>>>> had that
> >>>>> status for quite a while. I have created an overview of all these
> >>>>> tests to
> >>>>> visualize our current coverage:
> >>>>>
> >>>>> http://www.w3.org/2007/OWL/wiki/Test_Suite_Status#Proposed_tests
> >>>>>
> >>>>> Unfortunately, there are still some errors in the reports (e.g. I
> >>>>> have positive reports for CEL and FaCT++ for tests that are only
> >>>>> applicable under
> >>>>> RDF-based semantics). So some rows in that section are greener
> >>>>> than
> >>>>> they
> >>>>> should be, but most of them should be accurate.
> >>>>>
> >>>>> Many of the proposed tests are already covered by two or more
> >>>>> implementations,
> >>>>> and thus could probably be accepted easily. Since there is no next
> >>>>> telecon
> >>>>> scheduled so far, I hope we could do this via email -- I can
> >>>>> compile a list of
> >>>>> tests that seem to be ready.
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Markus
> >>>>>
> >>>>> [1] http://www.w3.org/2007/OWL/wiki/Test_Suite_Status
> >>>>>
> >>>>> --
> >>>>> Markus Krötzsch
> >>>>> Institute AIFB, Universität Karlsruhe (TH), 76128 Karlsruhe
> >>>>> phone +49 (0)721 608 7362           fax +49 (0)721 608 5998
> >>>>> mak@aifb.uni-karlsruhe.de           www  http://korrekt.org
> >>>>> http://semantic-web-book.org  http://semantic-mediawiki.org
> >
> > --
> > Markus Krötzsch
> > Institute AIFB, Universität Karlsruhe (TH), 76128 Karlsruhe
> > phone +49 (0)721 608 7362           fax +49 (0)721 608 5998
> > mak@aifb.uni-karlsruhe.de           www  http://korrekt.org
> > http://semantic-web-book.org  http://semantic-mediawiki.org


-- 
Markus Krötzsch
Institute AIFB, Universität Karlsruhe (TH), 76128 Karlsruhe
phone +49 (0)721 608 7362           fax +49 (0)721 608 5998
mak@aifb.uni-karlsruhe.de           www  http://korrekt.org
http://semantic-web-book.org  http://semantic-mediawiki.org