Re: TEST: proposal for test case structure

On Tue, 2002-07-16 at 06:39, Jeremy Carroll wrote:
> 
> 
> 
> Summary:
> - copy RDF Core
> - use directories per issue and per feature

hmm...

> - use boring file names that indicate test type and a number
> - use RDF/XML as normative, N-triple informative
> - have at least entailment, non-entailment, consistency, inconsistency and bad
> test types.
> - use manifest file to list tests and their status

ok.
(where is the RDF core manifest, btw? I don't recall seeing it.)

> - any WG member can create tests, but status is "PROPOSED"
> - delete/move/rename current test dir contents

ugh. Easier to leave it there and start over, but I hope
it doesn't come to that...

> I suggest we aim to copy RDF Core except where they appear to have made mistakes
> or where WebOnt goals differ.
> 
> This involves.
> 
> 1: Directories of directories of test cases.
> 
> I suggest the following:
> 
> http://www.w3.org/2002/03owlt/issues/
> 
> is a directory.

deep hierarchies are bad. I don't see any benefit
to another /issues/ level.

> For each issue with tests we construct a named subdirectory
> e.g. for 5.1
> 
> http://www.w3.org/2002/03owlt/issues/I5.1-Uniform-treatment-of-literal-data-valu
> es

I'd rather just

  http://www.w3.org/2002/03owlt/I5.1/

> We also have
> 
> http://www.w3.org/2002/03owlt/features/
> 
> as a directory.
> And for each language feature we have a sub-directory.

I'd rather just

  http://www.w3.org/2002/03owlt/Fxxx/

where xxx refers to a section of the language description document.


> (RDF Core only has issue driven tests - the structure above differs from theirs
> to explicitly allow for other tests)
> 
> 2: File locations
> 
> I suggest test files should only appear in leaf directories.

I'm OK with that for approved tests.


> 3: File types
> 
> I suggests all normative tests should be in RDF/XML.
> Each of these should have a non-normative N-triple analogue, produced
> automatically.
> (Or perhaps the method of production is the converse!)

So far, I've used N3 to draft most of the test cases.
Regardless, yes, RDF/XML is the only thing that I expect
we'll rely on.

> There should be no n3 in the test cases directory. (There currently is).

Why not? Why not n3 and UML/XMI and XML Schemas
and Makefiles and whatever else supports
the engineering of the tests?

> 4: Test types
> 
> We have currently approved tests that are:
> - entailment tests
> and
> - illegal use of OWL namespace
> 
> I expect that we will have consistency tests as well (this document is/is not
> self-consistent). These could be regarded as a special case of entailment.

I'm open to that.

> We may need other test types.
> 
> 4.1 Entailment test
> 
> A positive entailment test consists of two documents. One is the premises, the
> other is the conclusions.
> The conclusions follow from the premises.
> 
> A negative entailment test  consists of two documents. One is the premises, the
> other is the conclusions.
> The conclusions do not follow from the premises.

To be precise: the conclusion can't be justified from
the premises and the OWL spec alone. Let's not
make the closed-world assumption that it actually
doesn't follow with some other justification.

> 4.2 Illegal document test.
> 
> An illegal document test consists of one document.
> This document is not legal.
> A comment indicates the nature of the illegality.
> 
> 4.3 Consistency test
> 
> A positive consistency test consists of one document.
> The document is consistent.
> 
> A positive consistency test consists of one document.
> The document is inconsistent.

I would need to see some examples of these to be
convinced this is worth doing.


> 5: Test file naming
> 
> In RDF Core the names of the test files *do not* convey enough information to
> execute the test. (But nearly) A test manifest file is needed.
> 
> In RDF core nearly all the files are named
> 
> test0NN.rdf and test0NN.nt
> 
> where 0NN is a three digit number.
> 
> IMO, this has not worked well for entailment tests because of confusion about
> premises and conclusions.

yeah, I can see that.

> I suggest that for each test type there should be naming conventions for those
> files.
> 
> e.g.
> 
> 5.1 Entailment test
> 
> A positive entailment test consists of two documents.
> premises001.rdf
> conclusions001.rdf
> 
> If the converse entailment holds this is only recorded in the manifest.
> 
> A negative entailment test consists of two documents.
> premises001.rdf
> nonconclusions001.rdf

fair enough.


> 5.2 Illegal document test.
> 
> An illegal document test consists of one document.
> bad001.rdf

or even
	bad001.txt

lest the W3C webmaster be guilty of serving broken
stuff under the RDF mime type.

(we could make special Content-Type rules in our directory, but
I'd rather not)

> 5.3 Consistency test
> 
> A positive consistency test consists of one document.
> consistent001.rdf
> 
> A positive consistency test consists of one document.
> inconsistent001.rdf
> 
> 
> 6: Manifest file
> 
> Each directory has a manifest file describing the tests in it.
> RDF core has defined a schema. I suggest we use that with extensions.
> We may wish to make a richer description of the schema using OWL.
> 
> The manifest file may represent tests using the rdf files which are not apparant
> from the naming conventions. The most common will be two-way entailment
> (equivalent documents); it might be worth having a naming convention for those.
> 
> 7: File creating and modification
> 
> Proposed tests should be added by any WG member to the appropriate subdirectory.
> These should be added to the manifest with status="PROPOSED".
> 
> tests can be updated by their creator or someone acting on behalf of the WG, or
> any member of the TEST subgroup making editorial changes.
> 
> 
> 
> =====
> 
> An issue that appears is that the current test case area is already populated
> with files that do not adhere to these proposed conventions.
> 
> Choices are:
> - move/rename/delete those files

Deleting isn't an option. There are incoming links to those files
already.

Moving (i.e. leaving behind redirections) is an option.

> - start with a fresh empty test area

That's possible, or:

 - don't pay much attention to the files that don't conform to
	our conventions.

The latter is easiest for me.

 
-- 
Dan Connolly, W3C http://www.w3.org/People/Connolly/

Received on Wednesday, 17 July 2002 14:08:15 UTC