RE: Concerns regarding the W3 Document Object Model (DOM) Conform ance Test Suites

> - Offer of Help

Are appreciated.

> Success
> ----
> In order for statements of success when performing the 
> conformance tests 
> to be meaningful, the DOM TS must not include implementation 
> workarounds 
> which hide the inadequacies of particular implementations. 
> While it may 
> be desirable to allow the DOM TS to operate on an implementation in a 
> 'compatible' or 'equivalence' mode, such workarounds should 
> be clearly 
> labeled as not part of the actual Conformance test and must not be 
> allowed to hide any particular failure of an implementation 
> to conform 
> to the DOM Specification being tested.  These conformance 
> tests should 
> not allow functionally equivalent, non-standard proprietary 
> implementations to be passed off as conformant implementations of the 
> Standard being tested unless they are in fact conformant to 
> the letter 
> of the specification.

Definitely any number of sins can be hidden by the adaption layer.  The most
obvious "equivalence" kludge is the elimination of the ProcessingInstruction
node that is inserted by MSXML to represent the XML declaration.  The
resolution from the WG was that using a PI to represent the XML declaration
was not non-conformant.  Addressing this by removing the node in the
adaption layer for MSXML was much more expedient than adding a significant
amount of complexity to a large number of test definitions.

(I'll discuss the DOMException equivalence much later).

> 
> Coverage
> ----
> In order for a conformance test to be meaningful, the coverage of the 
> specification being tested by the conformance test must be 
> reported. If 
> the conformance test does not test aspects of the specification, this 
> should be made clear in the test documentation as well as in 
> the final 
> results of the conformance test.

The test matrix can be used to identify the lack of any test cases for a
particular function.  A converage analysis of a processor under test might
be used to identify code paths within that processor that are not being
executed and might suggest additional tests to add to the test suite.

> 
> 
> Concerns regarding jsUnit 1.2.6
> ====
> 
> jsUnit's Test Reports are not clear and unambiguous.
> 
> jsUnit's ECMAScript testRunner.html reports number of test 
> runs, number 
> of Errors and number of Failures and allows detailed reports of Error 
> and Failures to be produced.
> jsUnit's reported Errors are errors which occurred during the 
> execution 
> of the jsUnit test harness and do not reflect upon the conformance of 
> the implementation being tested. Any such errors should be clearly 
> reported as failures in the test harness and every effort 
> should be made 
> to eliminate any use of non-standard coding techniques or APIs which 
> cause such Errors. Under no circumstances should failures in the test 
> harness be attributed to the implementation's success or 
> failure of the 
> test case where the test harness error occurred.
> 
> The current format of jsUnit's reports does not clearly make this 
> distinction between Test Harness Errors and Conformance 
> Failures and can 
> lead to confusion as to the actual meaning of the result of a test.
> 
> jsUnit's reports should include not only failure reports but should 
> include a specific reference to the particular aspect of the 
> specification which failed the test. 
> 
> Test Reports should allow the user to determine how much of the 
> specification has tests defined for it, as well as which parts of the 
> specification were successfully or unsuccessfully implemented in 
> addition to which parts of the specification could not be tested.

JSUnit is a general purpose xUnit test framework for ECMAScript.  While the
maintainer of the JSUnit has been receptive to improvements that benefit all
JSUnit users, making changes that are highly specific to DOM TS is unlikely.

One obvious shortcoming with JSUnit is that tests that are not applicable to
the parser configuation are counted successfully completed tests though the
test code is never executed since there is no mechanism to distinguish
between an exception in the test constructor (indicating the test is not
applicable) with an exception in the body of the test.

It is clearer in the Java binding where dom-core.jar itself is independent
of any test suite and distinct adapters can be used to run the tests within
a particular testing framework.  On the java side, there were adapters for
both JUnit and Avalon testlet.  The Avalon testlet is currently out-of-date
since the Avalon testlet framework had been discontinued and the source code
snapshot used to build the adapter has been removed from the Apache servers.
However, it did work.

In a like manner, the tests generated by test-to-ecmascript.xsl could be run
by multiple testing frameworks and a testing framework could be developed to
address the issues you raised.

> 
> Concerns regarding Browser sniffing techniques used in the DOM TS ====
> 
> The current DOM TS performs browser sniffing for Netscape 
> 6/Mozilla and 
> Internet Explorer 5 and up. This limitation of the test suite to 
> particular vendor implementations is not necessary and is overly 
> restrictive.  Object sniffing can be used to admit any browser which 
> implements the relevant features of ECMAScript and the DOM 
> Specification 
> being tested.


All the current tests depend on loading a document to initialize the DOM,
since there is not a standard mechanism (yet) for loading an XML document,
therefore the adaptation layer must sniff which leaves the possibility that
an implementation that would be conformant cannot run the tests since it was
not recognized.

JSUnit will complain if the implementation is not recognized as Netscape
6/Mozilla or Internet Explorer 5.  I'm sure the maintainer of JSUnit would
appreciate any patches that allow JSUnit to run on other browsers.

> Browser/Vendor specific workarounds which hide particular 
> deficiencies 
> in an implementation in a "Compatible" or "Equivalence" mode may need 
> specific Browser/Vendor detection, however this should not 
> restrict the 
> ability of any browser with a conformant implementation of the DOM 
> Specification to attempt to run the DOM TS.
> 
> Concerns regarding use of APIs external to the DOM 
> Specification being 
> tested.
> ====
> 
> The current version of the DOM Core Level 1 tests rely upon the 
> Document.load method. This has three implications:

The tests do not rely on the Document.load method, they depend on the
adaption layer's implementation of load() (either as a global function in
ECMAScript or as a member of the super class in Java) to create a document
from the specified URL.

> 
> Implementations of the DOM Core Level 1 specification are 
> excluded from 
> using the DOM TS Core Level 1 if they do not implement this method.

The adaption layer must provide this function, however how it is
accomplished is insignificant to the test suite.

> The DOM TS uses external XML documents to create test instances of 
> Documents. See "Concerns regarding use of external Documents" 
> for more 
> on this issue.

It would be beneficial to have tests that build documents without loading
from an source document.  However, loading documents from a URL is the most
common use case and there must be tests that load from external documents to
be representative of that common use.

> 
> Implementation differences in the Document.load method cause 
> differences 
> in test results. Netscape 6/Mozilla's Document.load loads documents 
> asynchronously while Internet Explorer provides for either 
> asynchronous 
> or synchronous operation. As a result, test results can differ if the 
> 'alert' provided to Netscape 6/Mozilla is dismissed too 
> early. This type 
> of dependency is inappropriate in a conformance test.

Addressing this issue would require either Netscape 6/Mozilla to provide a
method to synchronously load an XML document or a testing framework that
supports asychronous tests.  Obviously either of these would be welcomed.


> 
> Concerns regarding use of external Documents
> ====
> 
> By creating test Documents through the use of external 
> documents and the 
> Parser, failures or limitations in the implementation of the 
> Parser are 
> inappropriately attributed to the implementation of the DOM 
> Specification being tested.

Obviously if the underlying parser drops characters or omits elements, the
DOM implementation cannot compensate and is not the source of the failure.
However, it is not reasonable for the test harness to determine if a
particular failure is the result of the parser implementation or the DOM
implementation, particularly since in some implementations, you cannot
distinguish between the layers.

> 
> In addition, several of the test documents use an external 
> DTD and fall 
> under the classification as standalone="no" documents since they use 
> default attribute values, etc. See "Extensible Markup 
> Language (XML) 1.0 
> (Second Edition) Section 2.9.
> 
> Section 5.2 "Using XML Processes" states:
> 
> In bullet two:
> 
> <quote>
> For example, a non-validating processor may not normalize [p.29] 
> attribute values, include [p.40] the replacement text of internal 
> entities, or supply default attribute values [p.28], where doing so 
> depends on having read declarations in external or parameter 
> entities. </quote>
> 
> In the final paragraph of this section:
> 
> <quote>
> Applications which require facilities such as the use of default 
> attributes or internal entities which are declared in 
> external entities 
> should use validating parsers.
> </quote>
> 
> Requiring the use of a validating Parser in the DOM TS is 
> inappropriate 
> since this is not required in the DOM Specifications themselves.

The test suite does not require the use of validating parsers.  Any test
that is only applicable to validating parsers can be marked so that it
omitted when run with a non-validating parser (or a parser that supports
validation but with it disabled).

There are many tests that depend on the reading of DTD's for default
attribute values.  The common interpretation has been that providing default
attribute values is not optional even when the parser is not validating and
many implementations pass these tests when not validating.  If the DOM WG
states that that behavior is optional, then the tests that depended on that
optional behavior can be marked as conditional.

> 
> Concerns regarding Browser/Vendor specific workarounds
> ====
> 
> The current DOM TS provides work arounds for specific implementations 
> which do not adequately implement the DOM Specification.  These 
> workarounds allow the implementation to 'appear' to pass the 
> conformance 
> tests even though in actuality they do not.
> 
> Example - document.implementation
> ----
> Internet Explorer does not implement the document.implementation 
> interface of the DOM Core Level 1 specification however the DOM TS 
> provides a work around which creates test documents via a specific 
> version of the  MSXML ActiveX control.

The lack of document.implementation is reported as a failure of a particular
test case (probably named documentimplementation) for MSXML.  Having MSXML
fail all tests due to the lack of this one method is inappropriate.  Since
DOM L1 does not provide any standard mechanism for loading, proprietary
methods are used to load documents.  Even when DOM L3 Load/Save is a
recommendation, failure to implement it would not mean that the
implementation is not conformant with L1 Core, it would just be
non-conformant with DOM L3 Load/Save.

> 
> Example - DOMException
> ----
> Internet Explorer/MSXML do not implement the DOMException interface 
> however the DOM TS provides a mapping from Internet Explorer/MSXML's 
> proprietary exceptions into the DOMException specification and hides 
> Internet Explorer's lack of conformance.

This is a reasonable cop and should be clearly labelled.  MSXML is a COM
parser and Microsoft's COM/Scripting interoperability layer provides no
mechanism (at least as far as I can determine) for a COM component to throw
an exception with a code attribute.  The approach of the current adaption
layer mimics the original NIST ECMAScript testing framework which was
forgiving of this deviation (actually it required everybody else to conform
with this deviation).

That said any COM implementation (MSXML, Apache-C's COM wrapper and Adobe
SVG's control, for example) accessed from Microsoft's implementation of
ECMAScript CANNOT be compliant with the ECMAScript binding's definition of
DOMException.

It would preferred at least that MSXML use HRESULT's that had a direct
correspondance with the DOMException.code values.  I believe that Adobe SVG
and the Apache-C COM wrapper do have direct mapping between the values (you
have to mask off the high bits to get the DOMException.code value).

> While providing a 'compatible' or 'equivalence' mode for tests for an 
> implementation in order to provide as much information to the 
> implementation's developers as well as users of the implementation is 
> appropriate and should be included in the DOM TS, any such 
> workarounds 
> should be clearly delineated from the actual Conformance Tests and be 
> clearly labeled so as to not mislead users of the Conformance 
> tests as 
> to what is or is not a Standard.


The equivalence is in the adaption layer and not the tests.

> 
> All implementations should be judged equally with regard to 
> conformance 
> and not have the issues involved confused by such workarounds.
> 
> Concerns over the Coverage of the Specification by the DOM TS ====
> 
> While investigating the code contained in the ECMAScript file 
> DOMTestCase.js, it became clear that the method 
> Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did 
> not test 
> the reported exception but rather always returned true. The same was 
> true for the ASVDocumentBuilder as well. As already mentioned the 
> MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test 
> the actual 
> reported exception but instead mapped the proprietary 
> exception onto the 
> DOMException code values.

Leaving MozXMLDocumentBuilder_isDOMExceptionCode() always returning true is
an oversight.  I'll check changing the implementation of that function and
if ex.code == code, then Mozilla will pass without any asterisks.

Given the limitations of Microsoft's Scripting's integration with COM, there
is no mechanism for Adobe could raise a conformant exception.  I did suggest
that they use HRESULT's that have a direct correspondance with the
DOMException.code which would be the best they could do under the
circumstance.  I don't know if they could do better as a Netscape plug-in.


> This has lead to the question "How much of the DOM Core Level 1 
> Specification is actually tested by the DOM TS?"
> 
> The DOM TS should completely cover the DOM Specification being tested 
> and all test cases should adequately test every 
> implementation for the 
> DOM TS to have any real meaning.
> 
> Recommendations
> ====
> 
> jsUnit Errors while executing the DOM TS should be 
> investigated and any 
> implementation defects in jsUnit leading to these errors should be 
> eliminated to the greatest extent possible.
> 
> jsUnit Errors should be clearly labeled as failures in the 
> jsUnit test 
> harness and not as conformance failures of the implementation 
> being tested.

Any issues with JSUnit or the framework adapters should be quickly addressed
when identified.  However, the DOM TS group did not exhaustively go through
every test failure for a specific processor to ensure that everyone was
actually due to a problem in the implementation.

> 
> Conformance Test Reports should include reports of coverage of the 
> Specification being tested.

Any suggestions for improvement of the test matrix or the metadata within
the tests used to generate the test matrix are welcome.

> 
> For the basic Conformance tests, specific sniffing for particular 
> browsers should be eliminated and only "object oriented" feature 
> detection used in order to maximize the implementations which can be 
> tested via the DOM TS as well as increase the fairness of the 
> Conformance tests.

DOM L3 load/save will improve the situation on the ECMAScript front.
Currently any JAXP parser can be tested on the Java implementation with no
implementation specific adaption.

> If needed, separate "Compatible" or "Equivalence" test cases can be 
> provided to provide additional testing for implementations 
> which can not 
> perform the basic Conformance tests natively.

> The use of external Documents, parsers, the implicit requirement of a 
> validating parser and the use of Document.load should be eliminated. 
> This can be achieved by using the implementation of the DOM itself to 
> produce test documents rather than loading external 
> documents. The use 
> of Documents created by the implementation itself removes any 
> possibility of quirks or limitations in the Document.load 
> method or the 
> parser inappropriately being flagged as a failure in the DOM 
> implementation.

Eliminating the tests from loaded documents would leave untested a major use
case for a DOM implementation.  Most users of a DOM implementation would not
be satisfied if they knew they had a non-conformant experience but knew it
was the underlying XML parser and not the DOM implementation that was at
fault.

As said before, the test suite does not require a validating parser.  If you
believe that, for example, providing default attribute values is an optional
behavior then you can raise that with the Working Group.


> Browser/Vendor specific work arounds should be eliminated from the 
> Conformance tests and restricted to well identified "Compatible" or 
> "Equivalence" tests.

I agree there should be a disclaimer for DOMException's on the MSXML and
Adobe SVG Control.

> 
> The test cases must provide complete coverage of the 
> specification being 
> tested

The only effective way to approach complete coverage is to look at a
coverage analysis of the implementation under tests and identifying test
cases that would exercise those paths.  If anyone wants to submit tests that
improve the coverage, please feel free.

> and the test results mapped back into the 
> specification so that 
> the test results can be understood in relation to the 
> specification and 
> not the DOM TS.

Each test contains URL's identifying sections of the recommendation
addressed by the test.

> 
> Offer of Help
> ====
> 
> I would be willing to help with the DOM TS and believe that several 
> members of the mozilla.org community would be willing to help as well.
> 
> I would like to thank everyone who has worked on the DOM TS 
> and hope we 
> can work together.
> 
> Bob Clary
> 
> -- 
> Bob Clary, bclary@netscape.com
> Technology Evangelist, Netscape Communications
> 
>

Received on Thursday, 21 February 2002 21:05:54 UTC