Concerns regarding the W3 Document Object Model (DOM) Conformance Test Suites

Concerns regarding the W3 Document Object Model (DOM) Conformance Test 
Suites

Contents
====

- Scope of this message
- Meaning and Implication of Conformance
  - Failure
  - Success
  - Coverage
- Concerns regarding jsUnit 1.2.6
- Concerns regarding Browser sniffing techniques used in the DOM TS
- Concerns regarding use of APIs external to the DOM Specification being 
tested.
- Concerns regarding use of external Documents
- Concerns regarding Browser/Vendor specific workarounds
  - Example - document.implementation
  - Example - DOMException
- Concerns over the Coverage of the Specification by the DOM TS
- Recommendations
- Offer of Help

Scope of this message
====

This message outlines some of the concerns that we have regarding the 
current release of the DOM TS for DOM Core Level 1 as well as future 
releases of the DOM TS for other DOM specifications which may also share 
similarities with the current DOM TS for DOM Core Level 1.

Meaning and Implication of Conformance
====

According to: 
<http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115#conformance>

<quote>
The DOM TS aims, as explained above, at helping implementers to write 
applications that support the DOM specifications. In no way are these 
conformance tests in the sense of providing companies or institutions 
with certification of DOM support. The only claim that could be made is 
that a particular implementation is conformant to a particular version 
of the DOM TS. There are two cases of results of running the test suite:

   1. The implementation fails to pass the test suite. In this case it 
can be asserted that the implementation fails to meet the relevant DOM 
specification.
   2. The implementation passes the test suite. In this case all that 
can be asserted is that the implementation is conformant to that 
particular version of the DOM TS.
</quote>

While the DOM TS is not intended to provide a certification process for 
an implementer, statements #1 and #2 above are quite strong and have a 
clear meaning.

  Failing to pass the DOM TS implies failure to implement the DOM 
Specification.

  Passing the DOM TS implies success in implementing the DOM Specification.

Failure
----
In order for statements of failure when performing the conformance test 
to be meaningful, the DOM TS must only report upon failures of 
conformance to the relevant DOM Specification being tested. It is 
important that failures which are a result of the Test Harness design or 
auxiliary APIs not be attributed to the implementation of the DOM 
Specification being tested. While failures in the Test Harness should be 
reported, they should not be attributed to failure to implement the DOM 
Specification being tested. The use of auxiliary APIs should be limited 
to those cases where it is absolutely necessary for the operation of the 
Test Harness and failures due to auxiliary APIs should not be attributed 
to failure to implement the DOM Specification being tested.

Success
----
In order for statements of success when performing the conformance tests 
to be meaningful, the DOM TS must not include implementation workarounds 
which hide the inadequacies of particular implementations. While it may 
be desirable to allow the DOM TS to operate on an implementation in a 
'compatible' or 'equivalence' mode, such workarounds should be clearly 
labeled as not part of the actual Conformance test and must not be 
allowed to hide any particular failure of an implementation to conform 
to the DOM Specification being tested.  These conformance tests should 
not allow functionally equivalent, non-standard proprietary 
implementations to be passed off as conformant implementations of the 
Standard being tested unless they are in fact conformant to the letter 
of the specification.

Coverage
----
In order for a conformance test to be meaningful, the coverage of the 
specification being tested by the conformance test must be reported. If 
the conformance test does not test aspects of the specification, this 
should be made clear in the test documentation as well as in the final 
results of the conformance test.


Concerns regarding jsUnit 1.2.6
====

jsUnit's Test Reports are not clear and unambiguous.

jsUnit's ECMAScript testRunner.html reports number of test runs, number 
of Errors and number of Failures and allows detailed reports of Error 
and Failures to be produced.
jsUnit's reported Errors are errors which occurred during the execution 
of the jsUnit test harness and do not reflect upon the conformance of 
the implementation being tested. Any such errors should be clearly 
reported as failures in the test harness and every effort should be made 
to eliminate any use of non-standard coding techniques or APIs which 
cause such Errors. Under no circumstances should failures in the test 
harness be attributed to the implementation's success or failure of the 
test case where the test harness error occurred.

The current format of jsUnit's reports does not clearly make this 
distinction between Test Harness Errors and Conformance Failures and can 
lead to confusion as to the actual meaning of the result of a test.

jsUnit's reports should include not only failure reports but should 
include a specific reference to the particular aspect of the 
specification which failed the test. 

Test Reports should allow the user to determine how much of the 
specification has tests defined for it, as well as which parts of the 
specification were successfully or unsuccessfully implemented in 
addition to which parts of the specification could not be tested.

Concerns regarding Browser sniffing techniques used in the DOM TS
====

The current DOM TS performs browser sniffing for Netscape 6/Mozilla and 
Internet Explorer 5 and up. This limitation of the test suite to 
particular vendor implementations is not necessary and is overly 
restrictive.  Object sniffing can be used to admit any browser which 
implements the relevant features of ECMAScript and the DOM Specification 
being tested.

Browser/Vendor specific workarounds which hide particular deficiencies 
in an implementation in a "Compatible" or "Equivalence" mode may need 
specific Browser/Vendor detection, however this should not restrict the 
ability of any browser with a conformant implementation of the DOM 
Specification to attempt to run the DOM TS.

Concerns regarding use of APIs external to the DOM Specification being 
tested.
====

The current version of the DOM Core Level 1 tests rely upon the 
Document.load method. This has three implications:

Implementations of the DOM Core Level 1 specification are excluded from 
using the DOM TS Core Level 1 if they do not implement this method.

The DOM TS uses external XML documents to create test instances of 
Documents. See "Concerns regarding use of external Documents" for more 
on this issue.

Implementation differences in the Document.load method cause differences 
in test results. Netscape 6/Mozilla's Document.load loads documents 
asynchronously while Internet Explorer provides for either asynchronous 
or synchronous operation. As a result, test results can differ if the 
'alert' provided to Netscape 6/Mozilla is dismissed too early. This type 
of dependency is inappropriate in a conformance test.

Concerns regarding use of external Documents
====

By creating test Documents through the use of external documents and the 
Parser, failures or limitations in the implementation of the Parser are 
inappropriately attributed to the implementation of the DOM 
Specification being tested.

In addition, several of the test documents use an external DTD and fall 
under the classification as standalone="no" documents since they use 
default attribute values, etc. See "Extensible Markup Language (XML) 1.0 
(Second Edition) Section 2.9.

Section 5.2 "Using XML Processes" states:

In bullet two:

<quote>
For example, a non-validating processor may not normalize [p.29] 
attribute values, include [p.40] the replacement text of internal 
entities, or supply default attribute values [p.28], where doing so 
depends on having read declarations in external or parameter entities.
</quote>

In the final paragraph of this section:

<quote>
Applications which require facilities such as the use of default 
attributes or internal entities which are declared in external entities 
should use validating parsers.
</quote>

Requiring the use of a validating Parser in the DOM TS is inappropriate 
since this is not required in the DOM Specifications themselves.

Concerns regarding Browser/Vendor specific workarounds
====

The current DOM TS provides work arounds for specific implementations 
which do not adequately implement the DOM Specification.  These 
workarounds allow the implementation to 'appear' to pass the conformance 
tests even though in actuality they do not.

Example - document.implementation
----
Internet Explorer does not implement the document.implementation 
interface of the DOM Core Level 1 specification however the DOM TS 
provides a work around which creates test documents via a specific 
version of the  MSXML ActiveX control.

Example - DOMException
----
Internet Explorer/MSXML do not implement the DOMException interface 
however the DOM TS provides a mapping from Internet Explorer/MSXML's 
proprietary exceptions into the DOMException specification and hides 
Internet Explorer's lack of conformance.

While providing a 'compatible' or 'equivalence' mode for tests for an 
implementation in order to provide as much information to the 
implementation's developers as well as users of the implementation is 
appropriate and should be included in the DOM TS, any such workarounds 
should be clearly delineated from the actual Conformance Tests and be 
clearly labeled so as to not mislead users of the Conformance tests as 
to what is or is not a Standard.

All implementations should be judged equally with regard to conformance 
and not have the issues involved confused by such workarounds.

Concerns over the Coverage of the Specification by the DOM TS
====

While investigating the code contained in the ECMAScript file 
DOMTestCase.js, it became clear that the method 
Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did not test 
the reported exception but rather always returned true. The same was 
true for the ASVDocumentBuilder as well. As already mentioned the 
MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test the actual 
reported exception but instead mapped the proprietary exception onto the 
DOMException code values.

This has lead to the question "How much of the DOM Core Level 1 
Specification is actually tested by the DOM TS?"

The DOM TS should completely cover the DOM Specification being tested 
and all test cases should adequately test every implementation for the 
DOM TS to have any real meaning.

Recommendations
====

jsUnit Errors while executing the DOM TS should be investigated and any 
implementation defects in jsUnit leading to these errors should be 
eliminated to the greatest extent possible.

jsUnit Errors should be clearly labeled as failures in the jsUnit test 
harness and not as conformance failures of the implementation being tested.

Conformance Test Reports should include reports of coverage of the 
Specification being tested.

For the basic Conformance tests, specific sniffing for particular 
browsers should be eliminated and only "object oriented" feature 
detection used in order to maximize the implementations which can be 
tested via the DOM TS as well as increase the fairness of the 
Conformance tests.

If needed, separate "Compatible" or "Equivalence" test cases can be 
provided to provide additional testing for implementations which can not 
perform the basic Conformance tests natively.

The use of external Documents, parsers, the implicit requirement of a 
validating parser and the use of Document.load should be eliminated. 
This can be achieved by using the implementation of the DOM itself to 
produce test documents rather than loading external documents. The use 
of Documents created by the implementation itself removes any 
possibility of quirks or limitations in the Document.load method or the 
parser inappropriately being flagged as a failure in the DOM implementation.

Browser/Vendor specific work arounds should be eliminated from the 
Conformance tests and restricted to well identified "Compatible" or 
"Equivalence" tests.

The test cases must provide complete coverage of the specification being 
tested and the test results mapped back into the specification so that 
the test results can be understood in relation to the specification and 
not the DOM TS.

Offer of Help
====

I would be willing to help with the DOM TS and believe that several 
members of the mozilla.org community would be willing to help as well.

I would like to thank everyone who has worked on the DOM TS and hope we 
can work together.

Bob Clary

-- 
Bob Clary, bclary@netscape.com
Technology Evangelist, Netscape Communications

Received on Thursday, 21 February 2002 18:10:47 UTC