RE: On conformance

Hi Christian, Felix, and all,

> So I think you should provide all tests which you 
> think which are necessary, not only the ones for 
> "terminology". This might be a very complicated task,
> *if* you assume a lot of conformance levels, and 
> even conformance specific conformance criteria to a 
> single data category.

Our data categories are quite divers: Ruby as little do to with translatability for example. This means it probably make sense for
the applications that will implement ITS to provide support for only some of the data categories.

For example a translation tool would implement the translatability data and localization information categories but completely
ignore terminology. Therefore I think we have to test the 6 data categories separately (I think <its:span> is something different
and can be tested with along with all the in situ cases).

>From the "rules location" viewpoint we have: in XML DTD, in XML Schema, in RELAX NG, external dislocated, internal dislocated, and
in situ... 6 cases. In addition, I think it's important to also have test cases for each data category where all the different
"rules locations" are combined. So 7 cases.

This gives us the following matrix:
http://www.w3.org/International/its/tests/#Summary

Which is ... 42 cases overall (although there maybe a few cases less as not all types of rules location apply to all data
categories).

I think it's important that we provide at least one standalone test case for each of these combinations. It is quite a bit of work,
but it is probably the only way to ensure ITS is sound. 

As far as "processors" *compliance*. I think we don't have to define a level for each case. Maybe we can say that an application is
ITS compliant when it implement sucessfully at least one of the data categories(?) and that it should state which one(s) with any
compliance claim.

We still have to decide if we want to allow processors that implement only in-situ rules to be compilant or not. We need to decide
this soon.

For the test cases, based on Felix and Christian's ideas, maybe we could have something for each data category that look like this:

1. In schema
	1.1 XML DTD
	1.2 XML Schema
	1.3 RELAX NG
2. Dislocated
	2.2 External to the document
	2.3 Within the document
3. In situ
4. Combination of all cases

For each of these lines we would have:

- The description of the test. (With a reference to the clause in the specification).

At least one test set that would have:

- An "Input files" entry with the list of all the input files required, for example a source XML document and a document containing
dislocated rules.

- An "Expected Result" entry with a document hand-made (or at least hand-checked) that describes the expected output.

- Zero, one or more result files generated from the various implementations we will have. (and hopefully will will have at least one
example of for each case).

See the translatability data category for an example:
http://www.w3.org/International/its/tests/#Trans_DislocatedExternal
(I'm missing still the clause references)

It would probably be good to have several test sets in some cases, for example; avec namespaces, without namespace, etc.

In addition to decide if this is a good approach and how it can be improved, we should also maybe make the general layout easier to
manipulate, for instance by having the Test Suite document broken down in several files (one per data category) so several people
can work on different parts at the same time. Maybe the result document should be integrated within the test suite document to make
it easier to look at, etc.

For the test implementation we should try to make them generic enough so they can be used regardless of the input files.

...I am sure you have plenty of ideas.

Cheers,
-yves

Received on Sunday, 12 February 2006 05:51:21 UTC