Re: Concerns regarding the W3 Document Object Model (DOM) Conform ance Test Suites

Most of these issues are sure to be brought up at the upcoming DOM WG 
F2F meeting next week in Cannes. In them meantime, as I'm travelling 
(first back home, then to Cannes) please provide the list with issues so 
that I can take them up with the WG.

comments inlined

On Friday, February 22, 2002, at 02:47 , Arnold, Curt wrote:

>> - Offer of Help
>
> Are appreciated.
>
>> Success
>> ----
>> In order for statements of success when performing the
>> conformance tests
>> to be meaningful, the DOM TS must not include implementation
>> workarounds
>> which hide the inadequacies of particular implementations.
>> While it may
>> be desirable to allow the DOM TS to operate on an implementation in a
>> 'compatible' or 'equivalence' mode, such workarounds should
>> be clearly
>> labeled as not part of the actual Conformance test and must not be
>> allowed to hide any particular failure of an implementation
>> to conform
>> to the DOM Specification being tested.  These conformance
>> tests should
>> not allow functionally equivalent, non-standard proprietary
>> implementations to be passed off as conformant implementations of the
>> Standard being tested unless they are in fact conformant to
>> the letter
>> of the specification.
>
> Definitely any number of sins can be hidden by the adaption layer.  The 
> most
> obvious "equivalence" kludge is the elimination of the 
> ProcessingInstruction
> node that is inserted by MSXML to represent the XML declaration.  The
> resolution from the WG was that using a PI to represent the XML 
> declaration
> was not non-conformant.  Addressing this by removing the node in the
> adaption layer for MSXML was much more expedient than adding a 
> significant
> amount of complexity to a large number of test definitions.
>
> (I'll discuss the DOMException equivalence much later).
>
>>
>> Coverage
>> ----
>> In order for a conformance test to be meaningful, the coverage of the
>> specification being tested by the conformance test must be
>> reported. If
>> the conformance test does not test aspects of the specification, this
>> should be made clear in the test documentation as well as in
>> the final
>> results of the conformance test.
>
> The test matrix can be used to identify the lack of any test cases for a
> particular function.  A converage analysis of a processor under test 
> might
> be used to identify code paths within that processor that are not being
> executed and might suggest additional tests to add to the test suite.
>
[dd] Further to this, the DOM TS Group continually checks the submitted 
tests for coverage and thinks up new tests that could be added. For 
reasons of time, however, and mainly due to me, we haven't been as good 
at that as we could have. This is one of the things that I like to 
improve for the future.

Let me sharpen Curt's claim by saying  that the critical point is to 
have more tests submitted.

>>
>>
>> Concerns regarding jsUnit 1.2.6
>> ====
>>
>> jsUnit's Test Reports are not clear and unambiguous.
>>
>> jsUnit's ECMAScript testRunner.html reports number of test
>> runs, number
>> of Errors and number of Failures and allows detailed reports of Error
>> and Failures to be produced.
>> jsUnit's reported Errors are errors which occurred during the
>> execution
>> of the jsUnit test harness and do not reflect upon the conformance of
>> the implementation being tested. Any such errors should be clearly
>> reported as failures in the test harness and every effort
>> should be made
>> to eliminate any use of non-standard coding techniques or APIs which
>> cause such Errors. Under no circumstances should failures in the test
>> harness be attributed to the implementation's success or
>> failure of the
>> test case where the test harness error occurred.
>>
>> The current format of jsUnit's reports does not clearly make this
>> distinction between Test Harness Errors and Conformance
>> Failures and can
>> lead to confusion as to the actual meaning of the result of a test.
>>
>> jsUnit's reports should include not only failure reports but should
>> include a specific reference to the particular aspect of the
>> specification which failed the test.
>>
>> Test Reports should allow the user to determine how much of the
>> specification has tests defined for it, as well as which parts of the
>> specification were successfully or unsuccessfully implemented in
>> addition to which parts of the specification could not be tested.
>
> JSUnit is a general purpose xUnit test framework for ECMAScript.  While 
> the
> maintainer of the JSUnit has been receptive to improvements that 
> benefit all
> JSUnit users, making changes that are highly specific to DOM TS is 
> unlikely.
>
[dd] As noted earlier, JsUnit was chosen for convenience and membership 
in the XUnit framework. I don't think anyone in this group would be 
reluctant to using another framework if need be, but we'd probably 
prefer that JsUnit be modified, especially since the JsUnit maintainer 
has shown great understanding of points we've brought to attention.

> One obvious shortcoming with JSUnit is that tests that are not 
> applicable to
> the parser configuation are counted successfully completed tests though 
> the
> test code is never executed since there is no mechanism to distinguish
> between an exception in the test constructor (indicating the test is not
> applicable) with an exception in the body of the test.
>
> It is clearer in the Java binding where dom-core.jar itself is 
> independent
> of any test suite and distinct adapters can be used to run the tests 
> within
> a particular testing framework.  On the java side, there were adapters 
> for
> both JUnit and Avalon testlet.  The Avalon testlet is currently 
> out-of-date
> since the Avalon testlet framework had been discontinued and the source 
> code
> snapshot used to build the adapter has been removed from the Apache 
> servers.
> However, it did work.
>
> In a like manner, the tests generated by test-to-ecmascript.xsl could 
> be run
> by multiple testing frameworks and a testing framework could be 
> developed to
> address the issues you raised.
>
[dd] I've noted earlier, in a reply to J. Brittsan of Microsoft (I'm 
currently travelling and on modem, I cannot look up the archives; it was 
about a week ago) that we could use this approach to allow for using the 
same source XML test descriptions but generate different code for 
running in different implementations, if that would solve our problem. 
We do want to keep the XML descriptions intact, so using different 
transforms is a simpler way than rewriting the entire testing framework.

>>
>> Concerns regarding Browser sniffing techniques used in the DOM TS ====
>>
>> The current DOM TS performs browser sniffing for Netscape
>> 6/Mozilla and
>> Internet Explorer 5 and up. This limitation of the test suite to
>> particular vendor implementations is not necessary and is overly
>> restrictive.  Object sniffing can be used to admit any browser which
>> implements the relevant features of ECMAScript and the DOM
>> Specification
>> being tested.
>
>
> All the current tests depend on loading a document to initialize the 
> DOM,
> since there is not a standard mechanism (yet) for loading an XML 
> document,
> therefore the adaptation layer must sniff which leaves the possibility 
> that
> an implementation that would be conformant cannot run the tests since 
> it was
> not recognized.
>
> JSUnit will complain if the implementation is not recognized as Netscape
> 6/Mozilla or Internet Explorer 5.  I'm sure the maintainer of JSUnit 
> would
> appreciate any patches that allow JSUnit to run on other browsers.
>
[dd] As far as I know, JsUnit is specifically constructed for the 
platforms you mention, according to the JsUnit homepage 
(http://www.jsunit.net)

>> Browser/Vendor specific workarounds which hide particular
>> deficiencies
>> in an implementation in a "Compatible" or "Equivalence" mode may need
>> specific Browser/Vendor detection, however this should not
>> restrict the
>> ability of any browser with a conformant implementation of the DOM
>> Specification to attempt to run the DOM TS.
>>
>> Concerns regarding use of APIs external to the DOM
>> Specification being
>> tested.
>> ====
>>
>> The current version of the DOM Core Level 1 tests rely upon the
>> Document.load method. This has three implications:
>
> The tests do not rely on the Document.load method, they depend on the
> adaption layer's implementation of load() (either as a global function 
> in
> ECMAScript or as a member of the super class in Java) to create a 
> document
> from the specified URL.
>
>>
>> Implementations of the DOM Core Level 1 specification are
>> excluded from
>> using the DOM TS Core Level 1 if they do not implement this method.
>
> The adaption layer must provide this function, however how it is
> accomplished is insignificant to the test suite.
>
>> The DOM TS uses external XML documents to create test instances of
>> Documents. See "Concerns regarding use of external Documents"
>> for more
>> on this issue.
>
> It would be beneficial to have tests that build documents without 
> loading
> from an source document.  However, loading documents from a URL is the 
> most
> common use case and there must be tests that load from external 
> documents to
> be representative of that common use.
>
>>
>> Implementation differences in the Document.load method cause
>> differences
>> in test results. Netscape 6/Mozilla's Document.load loads documents
>> asynchronously while Internet Explorer provides for either
>> asynchronous
>> or synchronous operation. As a result, test results can differ if the
>> 'alert' provided to Netscape 6/Mozilla is dismissed too
>> early. This type
>> of dependency is inappropriate in a conformance test.
>
> Addressing this issue would require either Netscape 6/Mozilla to 
> provide a
> method to synchronously load an XML document or a testing framework that
> supports asychronous tests.  Obviously either of these would be 
> welcomed.
>
[dd] As noted by myself, Mary and Curt, we've raised the issue a few 
times, obviously without solving it. Any help here is highly appreciated.

>
>>
>> Concerns regarding use of external Documents
>> ====
>>
>> By creating test Documents through the use of external
>> documents and the
>> Parser, failures or limitations in the implementation of the
>> Parser are
>> inappropriately attributed to the implementation of the DOM
>> Specification being tested.
>
> Obviously if the underlying parser drops characters or omits elements, 
> the
> DOM implementation cannot compensate and is not the source of the 
> failure.
> However, it is not reasonable for the test harness to determine if a
> particular failure is the result of the parser implementation or the DOM
> implementation, particularly since in some implementations, you cannot
> distinguish between the layers.
>
>>
>> In addition, several of the test documents use an external
>> DTD and fall
>> under the classification as standalone="no" documents since they use
>> default attribute values, etc. See "Extensible Markup
>> Language (XML) 1.0
>> (Second Edition) Section 2.9.
>>
>> Section 5.2 "Using XML Processes" states:
>>
>> In bullet two:
>>
>> <quote>
>> For example, a non-validating processor may not normalize [p.29]
>> attribute values, include [p.40] the replacement text of internal
>> entities, or supply default attribute values [p.28], where doing so
>> depends on having read declarations in external or parameter
>> entities. </quote>
>>
>> In the final paragraph of this section:
>>
>> <quote>
>> Applications which require facilities such as the use of default
>> attributes or internal entities which are declared in
>> external entities
>> should use validating parsers.
>> </quote>
>>
>> Requiring the use of a validating Parser in the DOM TS is
>> inappropriate
>> since this is not required in the DOM Specifications themselves.
>
> The test suite does not require the use of validating parsers.  Any test
> that is only applicable to validating parsers can be marked so that it
> omitted when run with a non-validating parser (or a parser that supports
> validation but with it disabled).
>
> There are many tests that depend on the reading of DTD's for default
> attribute values.  The common interpretation has been that providing 
> default
> attribute values is not optional even when the parser is not validating 
> and
> many implementations pass these tests when not validating.  If the DOM 
> WG
> states that that behavior is optional, then the tests that depended on 
> that
> optional behavior can be marked as conditional.
>
[dd] This is one of the issues that are sure to be brought up for 
clarification next week.
>>
>> Concerns regarding Browser/Vendor specific workarounds
>> ====
>>
>> The current DOM TS provides work arounds for specific implementations
>> which do not adequately implement the DOM Specification.  These
>> workarounds allow the implementation to 'appear' to pass the
>> conformance
>> tests even though in actuality they do not.
>>
>> Example - document.implementation
>> ----
>> Internet Explorer does not implement the document.implementation
>> interface of the DOM Core Level 1 specification however the DOM TS
>> provides a work around which creates test documents via a specific
>> version of the  MSXML ActiveX control.
>
> The lack of document.implementation is reported as a failure of a 
> particular
> test case (probably named documentimplementation) for MSXML.  Having 
> MSXML
> fail all tests due to the lack of this one method is inappropriate.  
> Since
> DOM L1 does not provide any standard mechanism for loading, proprietary
> methods are used to load documents.  Even when DOM L3 Load/Save is a
> recommendation, failure to implement it would not mean that the
> implementation is not conformant with L1 Core, it would just be
> non-conformant with DOM L3 Load/Save.
>
>>
>> Example - DOMException
>> ----
>> Internet Explorer/MSXML do not implement the DOMException interface
>> however the DOM TS provides a mapping from Internet Explorer/MSXML's
>> proprietary exceptions into the DOMException specification and hides
>> Internet Explorer's lack of conformance.
>
> This is a reasonable cop and should be clearly labelled.  MSXML is a COM
> parser and Microsoft's COM/Scripting interoperability layer provides no
> mechanism (at least as far as I can determine) for a COM component to 
> throw
> an exception with a code attribute.  The approach of the current 
> adaption
> layer mimics the original NIST ECMAScript testing framework which was
> forgiving of this deviation (actually it required everybody else to 
> conform
> with this deviation).
>
> That said any COM implementation (MSXML, Apache-C's COM wrapper and 
> Adobe
> SVG's control, for example) accessed from Microsoft's implementation of
> ECMAScript CANNOT be compliant with the ECMAScript binding's definition 
> of
> DOMException.
>
> It would preferred at least that MSXML use HRESULT's that had a direct
> correspondance with the DOMException.code values.  I believe that Adobe 
> SVG
> and the Apache-C COM wrapper do have direct mapping between the values 
> (you
> have to mask off the high bits to get the DOMException.code value).
>
>> While providing a 'compatible' or 'equivalence' mode for tests for an
>> implementation in order to provide as much information to the
>> implementation's developers as well as users of the implementation is
>> appropriate and should be included in the DOM TS, any such
>> workarounds
>> should be clearly delineated from the actual Conformance Tests and be
>> clearly labeled so as to not mislead users of the Conformance
>> tests as
>> to what is or is not a Standard.
>
>
> The equivalence is in the adaption layer and not the tests.
>
>>
>> All implementations should be judged equally with regard to
>> conformance
>> and not have the issues involved confused by such workarounds.
>>
>> Concerns over the Coverage of the Specification by the DOM TS ====
>>
>> While investigating the code contained in the ECMAScript file
>> DOMTestCase.js, it became clear that the method
>> Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did
>> not test
>> the reported exception but rather always returned true. The same was
>> true for the ASVDocumentBuilder as well. As already mentioned the
>> MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test
>> the actual
>> reported exception but instead mapped the proprietary
>> exception onto the
>> DOMException code values.
>
> Leaving MozXMLDocumentBuilder_isDOMExceptionCode() always returning 
> true is
> an oversight.  I'll check changing the implementation of that function 
> and
> if ex.code == code, then Mozilla will pass without any asterisks.
>
> Given the limitations of Microsoft's Scripting's integration with COM, 
> there
> is no mechanism for Adobe could raise a conformant exception.  I did 
> suggest
> that they use HRESULT's that have a direct correspondance with the
> DOMException.code which would be the best they could do under the
> circumstance.  I don't know if they could do better as a Netscape 
> plug-in.
>
>
>> This has lead to the question "How much of the DOM Core Level 1
>> Specification is actually tested by the DOM TS?"
>>
>> The DOM TS should completely cover the DOM Specification being tested
>> and all test cases should adequately test every
>> implementation for the
>> DOM TS to have any real meaning.
>>
>> Recommendations
>> ====
>>
>> jsUnit Errors while executing the DOM TS should be
>> investigated and any
>> implementation defects in jsUnit leading to these errors should be
>> eliminated to the greatest extent possible.
>>
>> jsUnit Errors should be clearly labeled as failures in the
>> jsUnit test
>> harness and not as conformance failures of the implementation
>> being tested.
>
> Any issues with JSUnit or the framework adapters should be quickly 
> addressed
> when identified.  However, the DOM TS group did not exhaustively go 
> through
> every test failure for a specific processor to ensure that everyone was
> actually due to a problem in the implementation.
>
>>
>> Conformance Test Reports should include reports of coverage of the
>> Specification being tested.
>
> Any suggestions for improvement of the test matrix or the metadata 
> within
> the tests used to generate the test matrix are welcome.
>
>>
>> For the basic Conformance tests, specific sniffing for particular
>> browsers should be eliminated and only "object oriented" feature
>> detection used in order to maximize the implementations which can be
>> tested via the DOM TS as well as increase the fairness of the
>> Conformance tests.
>
> DOM L3 load/save will improve the situation on the ECMAScript front.
> Currently any JAXP parser can be tested on the Java implementation with 
> no
> implementation specific adaption.
>
>> If needed, separate "Compatible" or "Equivalence" test cases can be
>> provided to provide additional testing for implementations
>> which can not
>> perform the basic Conformance tests natively.
>
>> The use of external Documents, parsers, the implicit requirement of a
>> validating parser and the use of Document.load should be eliminated.
>> This can be achieved by using the implementation of the DOM itself to
>> produce test documents rather than loading external
>> documents. The use
>> of Documents created by the implementation itself removes any
>> possibility of quirks or limitations in the Document.load
>> method or the
>> parser inappropriately being flagged as a failure in the DOM
>> implementation.
>
> Eliminating the tests from loaded documents would leave untested a 
> major use
> case for a DOM implementation.  Most users of a DOM implementation 
> would not
> be satisfied if they knew they had a non-conformant experience but knew 
> it
> was the underlying XML parser and not the DOM implementation that was at
> fault.
>
> As said before, the test suite does not require a validating parser.  
> If you
> believe that, for example, providing default attribute values is an 
> optional
> behavior then you can raise that with the Working Group.
>
>
>> Browser/Vendor specific work arounds should be eliminated from the
>> Conformance tests and restricted to well identified "Compatible" or
>> "Equivalence" tests.
>
> I agree there should be a disclaimer for DOMException's on the MSXML and
> Adobe SVG Control.
>
>>
>> The test cases must provide complete coverage of the
>> specification being
>> tested
>
> The only effective way to approach complete coverage is to look at a
> coverage analysis of the implementation under tests and identifying test
> cases that would exercise those paths.  If anyone wants to submit tests 
> that
> improve the coverage, please feel free.
>
>> and the test results mapped back into the
>> specification so that
>> the test results can be understood in relation to the
>> specification and
>> not the DOM TS.
>
> Each test contains URL's identifying sections of the recommendation
> addressed by the test.
>
>>
>> Offer of Help
>> ====
>>
>> I would be willing to help with the DOM TS and believe that several
>> members of the mozilla.org community would be willing to help as well.
>>
>> I would like to thank everyone who has worked on the DOM TS
>> and hope we
>> can work together.
>>
>> Bob Clary
>>
>> --
>> Bob Clary, bclary@netscape.com
>> Technology Evangelist, Netscape Communications
>>
>>
>

Received on Friday, 22 February 2002 06:10:51 UTC