RE: On conformance

Hi Felix,

This is great work. Nevertheless, here are some high-level comments ...

I have got the feeling that we currently do not clearly destinguish 

1. conformance clause
2. conformance level
3. conformance testing
4. test suite

Here's some prose I would use to talk about these concepts (cf.
http://www.itl.nist.gov/div897/ctg/conformance/bulletin-conformance.htm)
:

1. conformance clause: section in our a document that states which
requirement has to be met (e.g. Sec. 7.1.)
2. conformance level: a label attached to a set of conformance clauses
3. conformance testing: verify whether sth. is conformant (e.g. by
running an application on a test suite)
4. test suite: used to check whether results produced by an
implementation match the expected results

Based on these concepts, I would say that what we are currently aiming
at is a test suite.
Our test suite should provide an initial set of metrics to determine
whether or not sth. is
conformant (cf. http://www.w3.org/XML/Test/xmlconf-20031210.html).

For our test suite I suggest that that each test

- has an idenfitier
- relates to a requirement/conformance clause
- describes its purpose in prose
- is classified as binary or output
- lists the result which is expected from a conformant
application/document
(e.g. the string "rejected since ITS markup invalid" or an XML file with
the contents which should correctly by
classified as "to be translated")

Felix might know whether we need some kind of official statement that
tells people how the test suite is developed/maintained (cf.
http://www.w3.org/XML/Test/XMLConformanceTS-Process-20031210.html).

Coming back to my original "this is great work". From my understanding,
Felix already put the most important things
in place which would be needed to realize my suggestion. 

Best regards,
Christian

-----Original Message-----
From: public-i18n-its-request@w3.org
[mailto:public-i18n-its-request@w3.org] On Behalf Of Felix Sasaki
Sent: Mittwoch, 8. Februar 2006 14:32
To: public-i18n-its@w3.org
Subject: On conformance


Hi all,

This is on my action item (or the one of Christian and me) to have tests

for various conformance levels.


I) Testing a schema for conformance as described in sec. 7.1. Example:
<!ENTITY % itsmarkup SYSTEM "its.dtd">
%itsmarkup;
<!ELEMENT book ...>
<!ATTLIST book ... %att.datacats.attributes;>
each ATTLIST has the att.datacats.attributes entity.
Purpose: check the statement "The schema must allow the usage of the  
attribute group att.datacats at every element which is declared in the  
schema.".

- Testing a schema for conformance as described in sec. 7.2. Example:
<!ENTITY % itsmarkup SYSTEM "its.dtd">
%itsmarkup;
<!ELEMENT book (..., its:documentRules?>
<!ATTLIST book ... %att.datacats.attributes;>
Purpose: checksthe statement "The schema must allow the usage of the  
documentRules element in at least one element in the schema".

II) Testing interpretation of simple data category attributes as
described  
in sec. 7.1. Example:
<book ... its:translate="yes">
<head its:translate="no">...</head>
</book>
This document can be used to test the statement "The interpretation of  
data category attributes in instance documents must be conformant to the

data category specific default selections described in Section 4.1:  
Position and Default Selections of Data Categories." for the  
translatability data category. The implementations of Sebastian, Yves
and  
me would pass the tests because they select the nodes in accordance with

the in situ translatability data category definitions.


III) Testing Conformance to Dislocated Selection Mechanisms (see sec.  
7.2). Example:
<book ... its:translate="yes">
<head its:translate="no">
<its:documentRules>
<its:documentRule translate="yes" translateSelector="//p/@comment"/>
</its:documentRules>
...</head>
</book>
This document can be used to test the statement "An application which  
processes ITS elements and attributes must  process the selection  
mechanisms described in Section 3.5: Precedence between Selections". The

implementations of Sebastian, Yves and me would pass the tests because  
they select the nodes in accordance with the insitu and dislocated  
translatability data category definitions, and they take precedence of  
selection mechanisms into account.


Results of test:

- test type I: Result is a list of schemas which encompass the markup
for  
in situ usage of data categories. We already have enough schemas to pass

such tests, I would say. But it cannot hurt to have more.

- test type II and III: Results are "properly selected nodes".
"Properly"  
can be checked via visualization (as in the case of Sebastians  
implementation) or as a list of nodes (as in my case, or - I guess -
Yves  
case).

Looking forward for your feedback.

- Felix

Received on Thursday, 9 February 2006 17:16:41 UTC