Minutes from Informal Test related Telecon, 20/11/03

Minutes of Informal Test related Telecon, 20.11.03
=================================================================

Roll call: Ian Horrocks, Sean Bechhofer, Jeremy Carroll, Charles
White, Jos de Roo

1. Review of test status

Number of tests with only one pass. Probably due to people not
actually running the tests rather than a failure.

Racer? Off list exchange between JJC/JH/DC and RACER. No OWL test
harness or OWL support. Possibly Jena could use RACER through a DIG
harness. Possibly OWL API could use RACER through a DIG harness.

Could provide additional sets of test results by PR comments, which
would be sufficient.

Summary: constantly improving. Hoping for significant improvement in
the next week, but unlikely for next telecon.

[Action]: JJC contact implementors to tackle the easy/uncompleted
tests. Simple pass/fail is sufficient to allow us to approve the tests
at the next telecon. OWLP, Pellet, FaCT, Cerbera, Surnia, Euler. **

2: tests with 2 or more passes

AnnotationProperty-003 now Lite syntax. 2 passes for reasoners, 2
passes for syntax.

Put forward for approval by WG at next telecon.

3: OWL Full non-entailments and consistency tests

>  Class-004
>  DatatypeProperty-001
>  I5.1-010
>  I6.1-001

Not intended to be passed -- intended as extra credit
tests.

Propose we APPROVE these as EXTRACREDIT tests

4: tests from the extra-credit section
(one pass each)
  extra-credit-002
  extra-credit-003
  extra-credit-004

All positive entailment tests. Passed by Euler.

Propose that we APPROVE these as EXTRACREDIT tests

5: Tests with 0 passes

See earlier action (**) re. poking implementers. We believe a number
of these will be passed by two or more implementations, but people
haven't yet got round to running tests/reporting results.

6: Test with 1 pass

> Datatype tests
> I5.8-001
> I5.8-003
> I5.8-004

Propose that we APPROVE these as EXTRACREDIT tests as they use
datatypes that are not required to be implemented. They have been
passed by 1 implementation. We shouldn't have any tests outside extra
credit that require datatypes other than the required datatypes.

Jos: Some tests use int or nonNegative integer to specify the
cardinality constraints.

[Action]: JJC Check out which tests may use datatypes and thus should be
extra credit.

> easier (but new) tests
>    Thing-003
>    dl-208
>    misc-010
>    misc-011
>    Thing-005

See action (**) re. poking implementers.

>  hard dl tests
>    dl-668

668 depends on a translation of 208 which was incorrect.

>    dl-666

[Action]: JJC fix up 668 and provide two tests: one positive
entailment and one negative entailment.

We believe that we'll get results from RACER by the end of the
year. Leave these as PROPOSED -- we expect to APPROVE.

>    dl-501
>    dl-502

[Action]: CW to check out the results of these from Cerebra.
Propose APPROVE as EXTRACREDIT regardless.

>  arithmetic
>    dl-905 dl-906 dl-907 dl-909 dl-910

[Action]: CW to check out the results of these from Cerebra.

[Action]: JJC fix comment on dl-907

Euler data possibly not being interpreted correctly (no data rather
than undecided).

[Action]: JJC include dl-905 and dl-909 in the implementors message.

Awaiting more data re. dl-905,dl-906,dl-909, dl-901. Propose APPROVE
dl-907 as EXTRACREDIT.

> OWL Full tests with conflict
>  I5.3-014 *
>  someValuesFrom-001 *
>  Restriction-006 *
>  I5.5-007 *

These tests are failed by Pellet -- suggesting an unsound full
implementation.

[Action]: JJC Request Pellet to do a visual inspection
on the tests and report back anything dubious.

> OWL Full tests exhibiting some 'advanced' OWL Full features
>  I5.8-017

Quick eyeball suggests this is fine. Propose APPROVE I5.8-017

>  cardinality-005

Remain as PROPOSED.

7: Process issues

The test case document cannot really be regarded as finished when it
has a proposed tests appendix.

Ian: This is likely to be a permanent state.

JJC: What's the purpose of the test suite?

Ian: To provide confidence in implementations and illustrate design
decisions and issues.

SKB: Who's responsibility is it to curate and maintain the test suite?

Possible option: continue with PROPOSED tests. At the point when the
WG goes to REC, we are prepared to leave PROPOSED tests in. Does
process permit this? Do we need wording changes in the document to
reflect this?

[Action]: JJC compose words to DanC and Sandro regarding the above.

Summary
-------

PROPOSE that the following tests be APPROVED:

> AnnotationProperty-003
> I5.8-017

PROPOSE that the following tests be APPROVED as EXTRACREDIT:

> Class-004
> DatatypeProperty-001
> I5.1-010
> I6.1-001
> I5.8-001
> I5.8-003
> I5.8-004
> dl-501
> dl-502
> dl-907
> extra-credit-002
> extra-credit-003
> extra-credit-004

Actions:
--------

[Action]: JJC contact implementors to tackle the easy/uncompleted
tests. Simple pass/fail is sufficient to allow us to approve the tests
at the next telecon. OWLP, Pellet, FaCT, Cerbera, Surnia, Euler. **

[Action]: JJC include dl-905 and dl-909 in the above implementors message.

[Action]: JJC Check out which tests may use datatypes and thus should be
extra credit.

[Action]: CW to check out the results of dl-501 and dl-502 from
Cerebra.

[Action]: JJC fix comment on dl-907

[Action]: JJC Request Pellet to do a visual inspection on Tests with
conflict and report back anything dubious.

[Action]: JJC compose words to DanC and Sandro regarding process and
the possibility of leaving PROPOSED tests in at REC.



-- 
Sean Bechhofer
seanb@cs.man.ac.uk
http://www.cs.man.ac.uk/~seanb

Received on Wednesday, 26 November 2003 06:20:43 UTC