RE: Concerns regarding the W3 Document Object Model (DOM) Conformance Test Suites

Comments regarding DOMExceptions:

http://www.w3.org/TR/2000/WD-DOM-Level-1-20000929/level-one-core.html#ID
-17189187

<quote>
Some languages and object systems do not support the concept of
exceptions. For such systems, error conditions may be indicated using
native error reporting mechanisms. For some bindings, for example,
methods may return error codes similar to those listed in the
corresponding method descriptions.
</quote>

By returning error codes, Internet Explorer does conform to the DOM
Level 1 specification. 

-----Original Message-----
From: Dimitris Dimitriadis [mailto:dimitris@ontologicon.com] 
Sent: Thursday, February 21, 2002 4:33 PM
To: Robert Clary
Cc: www-dom-ts@w3.org
Subject: Re: Concerns regarding the W3 Document Object Model (DOM)
Conformance Test Suites

Thanks for your thoughts, Bob

Comments are inlined

/Dimitris Dimitriadis, W3C DOM WG representative to the DOM TS Group

On Friday, February 22, 2002, at 12:05 , Robert Clary wrote:

> Concerns regarding the W3 Document Object Model (DOM) Conformance Test

> Suites
>
> Contents
> ====
>
> - Scope of this message
> - Meaning and Implication of Conformance
>  - Failure
>  - Success
>  - Coverage
> - Concerns regarding jsUnit 1.2.6
> - Concerns regarding Browser sniffing techniques used in the DOM TS
> - Concerns regarding use of APIs external to the DOM Specification 
> being tested.
> - Concerns regarding use of external Documents
> - Concerns regarding Browser/Vendor specific workarounds
>  - Example - document.implementation
>  - Example - DOMException
> - Concerns over the Coverage of the Specification by the DOM TS
> - Recommendations
> - Offer of Help
>
> Scope of this message
> ====
>
> This message outlines some of the concerns that we have regarding the 
> current release of the DOM TS for DOM Core Level 1 as well as future 
> releases of the DOM TS for other DOM specifications which may also 
> share similarities with the current DOM TS for DOM Core Level 1.
>
> Meaning and Implication of Conformance
> ====
>
> According to: 
> <http://www.w3.org/2002/01/DOMConformanceTS-
> Process-20020115#conformance>
>
> <quote>
> The DOM TS aims, as explained above, at helping implementers to write 
> applications that support the DOM specifications. In no way are these 
> conformance tests in the sense of providing companies or institutions 
> with certification of DOM support. The only claim that could be made
is 
> that a particular implementation is conformant to a particular version

> of the DOM TS. There are two cases of results of running the test
suite:
>
>   1. The implementation fails to pass the test suite. In this case it 
> can be asserted that the implementation fails to meet the relevant DOM

> specification.
>   2. The implementation passes the test suite. In this case all that 
> can be asserted is that the implementation is conformant to that 
> particular version of the DOM TS.
> </quote>
>
> While the DOM TS is not intended to provide a certification process
for 
> an implementer, statements #1 and #2 above are quite strong and have a

> clear meaning.
>
[dd] Actually, this was anticipated, and the interpretation should 
rather be:

>  Failing to pass the DOM TS implies failure to implement the DOM 
> Specification.
>
[dd] Indeed, and to be exact, failing to pass the DOM TS implies failure

to implement the particular feature of the DOM Specification being 
tested, which in effect means failing to meet the relevant DOM 
Specification.

>  Passing the DOM TS implies success in implementing the DOM 
> Specification.
>
[dd] On rereading, what the second claim above amounts to is that the 
implementation, if it passes, passes the _test suite_ (a particular 
version of it, to be exact). So, no conformance with the DOM 
specification itself is warranted by passing the test suite.

> Failure
> ----
> In order for statements of failure when performing the conformance
test 
> to be meaningful, the DOM TS must only report upon failures of 
> conformance to the relevant DOM Specification being tested. It is 
> important that failures which are a result of the Test Harness design 
> or auxiliary APIs not be attributed to the implementation of the DOM 
> Specification being tested. While failures in the Test Harness should 
> be reported, they should not be attributed to failure to implement the

> DOM Specification being tested. The use of auxiliary APIs should be 
> limited to those cases where it is absolutely necessary for the 
> operation of the Test Harness and failures due to auxiliary APIs
should 
> not be attributed to failure to implement the DOM Specification being 
> tested.
>
[dd] Correct. Historically, JsUnit, which I believe is the part of the 
DOM TS framework that causes most concerns on your part, was chosen as a

convenience solution. In any case, errors that can be tracked to the 
framework and not the tests should be reported as such and not atributed

to the implementation tested.

> Success
> ----
> In order for statements of success when performing the conformance 
> tests to be meaningful, the DOM TS must not include implementation 
> workarounds which hide the inadequacies of particular implementations.

> While it may be desirable to allow the DOM TS to operate on an 
> implementation in a 'compatible' or 'equivalence' mode, such 
> workarounds should be clearly labeled as not part of the actual 
> Conformance test and must not be allowed to hide any particular
failure 
> of an implementation to conform to the DOM Specification being tested.

> These conformance tests should not allow functionally equivalent, 
> non-standard proprietary implementations to be passed off as
conformant 
> implementations of the Standard being tested unless they are in fact 
> conformant to the letter of the specification.
>
[dd] Also correct, in part. Here we enter a gray zone, as the DOM 
Specification only dictated what should be done, and not how. So, in 
those cases where it is obvious that the DOM specification is not 
followed, you're right. However, the DOM specification is flexible 
isnofar as it provides a wrapper around any implementation and is to be 
judged by behaviour.

> Coverage
> ----
> In order for a conformance test to be meaningful, the coverage of the 
> specification being tested by the conformance test must be reported.
If 
> the conformance test does not test aspects of the specification, this 
> should be made clear in the test documentation as well as in the final

> results of the conformance test.
>
[dd] This has been one of the group's largest concerns, and we have time

and again asked for interested parties to submit tests. The main bulk of

tests that NIST submitted were judged to be of enough coverage to 
release as a DOM L1 Core Test Suite. Any additional tests that increase 
coverage are highly appreciated.

>
> Concerns regarding jsUnit 1.2.6
> ====
>
> jsUnit's Test Reports are not clear and unambiguous.
>
> jsUnit's ECMAScript testRunner.html reports number of test runs,
number 
> of Errors and number of Failures and allows detailed reports of Error 
> and Failures to be produced.
> jsUnit's reported Errors are errors which occurred during the
execution 
> of the jsUnit test harness and do not reflect upon the conformance of 
> the implementation being tested. Any such errors should be clearly 
> reported as failures in the test harness and every effort should be 
> made to eliminate any use of non-standard coding techniques or APIs 
> which cause such Errors. Under no circumstances should failures in the

> test harness be attributed to the implementation's success or failure 
> of the test case where the test harness error occurred.
>
[dd] Where identifiable, I also believe we should try to state that it 
is the framework, not the implementation, that causes problems. Perhaps 
we should try to work a bit on the harness itself, then. I suppose 
similar views could be raised about the JUnit framework.

> The current format of jsUnit's reports does not clearly make this 
> distinction between Test Harness Errors and Conformance Failures and 
> can lead to confusion as to the actual meaning of the result of a
test.
>
> jsUnit's reports should include not only failure reports but should 
> include a specific reference to the particular aspect of the 
> specification which failed the test.
> Test Reports should allow the user to determine how much of the 
> specification has tests defined for it, as well as which parts of the 
> specification were successfully or unsuccessfully implemented in 
> addition to which parts of the specification could not be tested.
>
> Concerns regarding Browser sniffing techniques used in the DOM TS
> ====
>
> The current DOM TS performs browser sniffing for Netscape 6/Mozilla
and 
> Internet Explorer 5 and up. This limitation of the test suite to 
> particular vendor implementations is not necessary and is overly 
> restrictive.  Object sniffing can be used to admit any browser which 
> implements the relevant features of ECMAScript and the DOM 
> Specification being tested.
>
[dd] This is a known limitation in the JsUnit framework, but again, we 
chose it for convenience.

> Browser/Vendor specific workarounds which hide particular deficiencies

> in an implementation in a "Compatible" or "Equivalence" mode may need 
> specific Browser/Vendor detection, however this should not restrict
the 
> ability of any browser with a conformant implementation of the DOM 
> Specification to attempt to run the DOM TS.
>
> Concerns regarding use of APIs external to the DOM Specification being

> tested.
> ====
>
> The current version of the DOM Core Level 1 tests rely upon the 
> Document.load method. This has three implications:
>
> Implementations of the DOM Core Level 1 specification are excluded
from 
> using the DOM TS Core Level 1 if they do not implement this method.
>
> The DOM TS uses external XML documents to create test instances of 
> Documents. See "Concerns regarding use of external Documents" for more

> on this issue.
>
> Implementation differences in the Document.load method cause 
> differences in test results. Netscape 6/Mozilla's Document.load loads 
> documents asynchronously while Internet Explorer provides for either 
> asynchronous or synchronous operation. As a result, test results can 
> differ if the 'alert' provided to Netscape 6/Mozilla is dismissed too 
> early. This type of dependency is inappropriate in a conformance test.
[dd] We raised this issue (about the Mozilla loading) on several 
occasions without receiving feedback. I'm sure we could work aroud this 
and reach a commonly accepted solution in time for the next release, to 
which Netscape has commited to provide HTML tests.

> Concerns regarding use of external Documents
> ====
>
> By creating test Documents through the use of external documents and 
> the Parser, failures or limitations in the implementation of the
Parser 
> are inappropriately attributed to the implementation of the DOM 
> Specification being tested.
>
[dd] These limitations only occur at build time, not at run time. We 
could investigate to see if the software we've used produces errors that

are later attributed to the implementations tested by the DOM TS.

> In addition, several of the test documents use an external DTD and
fall 
> under the classification as standalone="no" documents since they use 
> default attribute values, etc. See "Extensible Markup Language (XML) 
> 1.0 (Second Edition) Section 2.9.
>
> Section 5.2 "Using XML Processes" states:
>
> In bullet two:
>
> <quote>
> For example, a non-validating processor may not normalize [p.29] 
> attribute values, include [p.40] the replacement text of internal 
> entities, or supply default attribute values [p.28], where doing so 
> depends on having read declarations in external or parameter entities.
> </quote>
>
> In the final paragraph of this section:
>
> <quote>
> Applications which require facilities such as the use of default 
> attributes or internal entities which are declared in external
entities 
> should use validating parsers.
> </quote>
>
> Requiring the use of a validating Parser in the DOM TS is
inappropriate 
> since this is not required in the DOM Specifications themselves.
>
> Concerns regarding Browser/Vendor specific workarounds
> ====
>
> The current DOM TS provides work arounds for specific implementations 
> which do not adequately implement the DOM Specification.  These 
> workarounds allow the implementation to 'appear' to pass the 
> conformance tests even though in actuality they do not.
>
> Example - document.implementation
> ----
> Internet Explorer does not implement the document.implementation 
> interface of the DOM Core Level 1 specification however the DOM TS 
> provides a work around which creates test documents via a specific 
> version of the  MSXML ActiveX control.
>
> Example - DOMException
> ----
> Internet Explorer/MSXML do not implement the DOMException interface 
> however the DOM TS provides a mapping from Internet Explorer/MSXML's 
> proprietary exceptions into the DOMException specification and hides 
> Internet Explorer's lack of conformance.
>
> While providing a 'compatible' or 'equivalence' mode for tests for an 
> implementation in order to provide as much information to the 
> implementation's developers as well as users of the implementation is 
> appropriate and should be included in the DOM TS, any such workarounds

> should be clearly delineated from the actual Conformance Tests and be 
> clearly labeled so as to not mislead users of the Conformance tests as

> to what is or is not a Standard.
>
> All implementations should be judged equally with regard to
conformance 
> and not have the issues involved confused by such workarounds.
>
[dd] I indicated that there can be grey zones above. However, where we 
identify such levels of difference from the DOM Specification, it should

certainly be discussed.

> Concerns over the Coverage of the Specification by the DOM TS
> ====
>
> While investigating the code contained in the ECMAScript file 
> DOMTestCase.js, it became clear that the method 
> Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did not test

> the reported exception but rather always returned true. The same was 
> true for the ASVDocumentBuilder as well. As already mentioned the 
> MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test the actual

> reported exception but instead mapped the proprietary exception onto 
> the DOMException code values.
>
> This has lead to the question "How much of the DOM Core Level 1 
> Specification is actually tested by the DOM TS?"
>
> The DOM TS should completely cover the DOM Specification being tested 
> and all test cases should adequately test every implementation for the

> DOM TS to have any real meaning.
>
[dd] Again, the group's intention was to provide as many tests as 
possible in order to achieve maximum coverage. For various reasons, 
mainly since no tests except for the initial NIST load were committed, 
we had to draw a line somewhere. Since we do not claim to have maximum 
coverage, however, future releases of the DOM TS will, especially if 
more companies commit tests, have better coverage.

> Recommendations
> ====
>
> jsUnit Errors while executing the DOM TS should be investigated and
any 
> implementation defects in jsUnit leading to these errors should be 
> eliminated to the greatest extent possible.
>
> jsUnit Errors should be clearly labeled as failures in the jsUnit test

> harness and not as conformance failures of the implementation being 
> tested.
>
> Conformance Test Reports should include reports of coverage of the 
> Specification being tested.
>
> For the basic Conformance tests, specific sniffing for particular 
> browsers should be eliminated and only "object oriented" feature 
> detection used in order to maximize the implementations which can be 
> tested via the DOM TS as well as increase the fairness of the 
> Conformance tests.
>
> If needed, separate "Compatible" or "Equivalence" test cases can be 
> provided to provide additional testing for implementations which can 
> not perform the basic Conformance tests natively.
>
> The use of external Documents, parsers, the implicit requirement of a 
> validating parser and the use of Document.load should be eliminated. 
> This can be achieved by using the implementation of the DOM itself to 
> produce test documents rather than loading external documents. The use

> of Documents created by the implementation itself removes any 
> possibility of quirks or limitations in the Document.load method or
the 
> parser inappropriately being flagged as a failure in the DOM 
> implementation.
>
> Browser/Vendor specific work arounds should be eliminated from the 
> Conformance tests and restricted to well identified "Compatible" or 
> "Equivalence" tests.
>
> The test cases must provide complete coverage of the specification 
> being tested and the test results mapped back into the specification
so 
> that the test results can be understood in relation to the 
> specification and not the DOM TS.
>
[dd] I agree with most of your points and think that they should be 
considered for implementation in the next versions of the DOM TS. 
However, it was anticipated quite early that we would not have a 
complete test suite, which is indicated in the DOM TS Process document 
that received acceptance from the DOM WG, IG and other related parties 
before being published:

<quote>
(point 2) The DOM test suites are intended to be used as a tool to aid 
implementors in developing DOM implementations. Validation and 
certification of these implementations are outside the scope of this 
work. The tests and test suites will be provided for information and 
help only. However, we intend to produce as comprehensive, functional 
and general tests as possible, and this should be the overall goal in 
designing and implementing the DOM TS.
</quote>

(http://www.w3.org/2002/01/DOMConformanceTS-
Process-20020115#requirements)

There is a clear intention to produce as comprehensive, functional and 
general tests as possible, but to claim is made that these will be 
complete.

> Offer of Help
> ====
>
> I would be willing to help with the DOM TS and believe that several 
> members of the mozilla.org community would be willing to help as well.
>
[dd] Your offer of help is wholeheartedly accepted and we look forward 
to having you on board.


> I would like to thank everyone who has worked on the DOM TS and hope
we 
> can work together.
>
[dd] Thanks for your many insights, I hope we'll have a good discussion 
going about the issues you raised.

> Bob Clary
>
> -- Bob Clary, bclary@netscape.com
> Technology Evangelist, Netscape Communications
>
>

Received on Thursday, 21 February 2002 19:45:46 UTC