Re: Open world value tests

On Tue, Oct 24, 2006 at 10:07:22AM +0100, Seaborne, Andy wrote:
> 
> 
> 
> 
> -------- Original Message --------
> > From: Steve Harris <>
> > Date: 24 October 2006 09:34
> > 
> > On 23 Oct 2006, at 23:29, Eric Prud'hommeaux wrote:
> > 
> > > > > 
> > > > > These tests bring up the issue of how we test extensions. I
> think
> > > > > all we can do is test the bare language, writing tests that
> appear
> > > > > to label an extended implementation as "failed".
> > > > 
> > > > Right - these tests were written to capture "="/"!=" on both known
> > > > and unknown types, and what happens when an extension datatype is
> > > > known.  Not all the tests will be appropriate in the code DAWG
> test
> > > > suite. 
> > > > 
> > > > For testing extensions, how about separating the tests out into a
> > > > separate area?  At least, have different manifest files so that an
> > > > implementation can pick up the manifests and run the appropriate
> > > > ones.
> > > 
> > > I prefer labeling these tests in the manifest, ala:
> > > 
> > >         [ mf:name    "date-1" ;
> > > 	rdfs:comment	"Added type : xsd:date '='" ;
> > >            mf:action
> > >               [ qt:query  <date-1.rq> ;
> > >                 qt:data   <data-3.ttl> ] ;
> > >         mf:result  <date-1-result.srx> ;
> > > 	mf:requires xsd:date
> > 
> > +1
> 
> -1
> 
> Tests don't have URLs; manifests do have URLs.  Manifests are a unit of
> organisation of tests.

Sorry, I don't follow this.

> Having manifests that an implementer can't just pick up give to their
> test framework makes their life harder.  Are they testing with or
> without feature X?  And ours too - we will need to provide a set of
> tests we wish to get implementation reports on.  Pointing to manifests
> by URL is direct and simple.  
> 
> Pointing to one manifest that includes the relevant ones is even easier
> and that will reduce our workload overall.
> 
> Extra vocabulary might also be confusing: an implementer might well say
> "my system passed all the tests in manifest X" but not notice that there
> were some optional and some base tests.

I think it's safe to assume that anyone running the tests is loading
the manifest and executing the tests listed in the manifest rather
than searching the directory. Thus, I feel it's more reasonable to
make the test charactersitics processable in RDF.

We could try to additionally sequester the sets of tests that require
a particular type of extension, for instance, xsd:date or
geo:distance, into separate directories, but I think that any
non-extended tests, or tests requiring different extensions, could not
go into that directory. If we want to process exclusively by
directory, we couldn't expect a processor to know it should process
open-eq-0[1-6] and not the ones requiring support for xsd:date .
Ultimately, I feel strongly that the authoritative data should be in RDF,
not in directory names.

Right now, my algae tester gets one set of results for

      [ mf:name    "date-1" ;
	rdfs:comment	"Added type : xsd:date '='" ;
           mf:action
              [ qt:query  <date-1.rq> ;
                qt:data   <data-3.ttl> ] ;
        mf:result  <date-1-result.srx> ;
	mf:requires xsd:date ]

and another set for

      [ mf:name    "date-1" ;
	rdfs:comment	"Added type : xsd:date '='" ;
           mf:action
              [ qt:query  <date-1.rq> ;
                qt:data   <data-3.ttl> ] ;
        mf:result  <date-no-date-1-result.srx> ]

> PS Processing that style in SPARQL would require testing for the absense
> of a triple pattern involving mf:requires :-)

No more than qt:data . As I find qt:data actions, I add the data to my
database. As I find mf:requires, I enable support for that requirement.

-- 
-eric

home-office: +1.617.395.1213 (usually 900-2300 CET)
	    +33.1.45.35.62.14
cell:       +33.6.73.84.87.26

(eric@w3.org)
Feel free to forward this message to any list for any purpose other than
email address distribution.

Received on Tuesday, 24 October 2006 14:13:29 UTC