Re: Concerns regarding the W3 Document Object Model (DOM) Conformance Test Suites

Dimitris,

Thank you for the prompt reply.

I will try to help with regard to the jsUnit test frame work and other 
issues related to scripting as much as possible.

Hopefully as time progresses the coverage of the test suites will 
increase as more contributions are made.

I understand some of the tradeoffs that have been made in developing the 
tests however these tests can easily be misinterpreted which has lead to 
incorrect statements regarding Mozilla's support for the W3C DOM. 
 Whatever the limitations that you have been working under it is 
incorrect to allow an incomplete test suite which itself does not claim 
to test conformance to be represented as such.

Additional comments inline

Dimitris Dimitriadis wrote:

[snip]

>>
> [dd] On rereading, what the second claim above amounts to is that the 
> implementation, if it passes, passes the _test suite_ (a particular 
> version of it, to be exact). So, no conformance with the DOM 
> specification itself is warranted by passing the test suite.

[bc] I can see your point with regard to the quoted text however this is 
a very fine distinction to make especially  where the title of 
http://www.w3.org/DOM/Test/ is "Document Object Model (DOM) Conformance 
Test Suites". Perhaps these should be renamed to "Document Object Model 
(DOM)  Test Suites" to remove the possibilitiy of misunderstanding.

> [dd] Also correct, in part. Here we enter a gray zone, as the DOM 
> Specification only dictated what should be done, and not how. So, in 
> those cases where it is obvious that the DOM specification is not 
> followed, you're right. However, the DOM specification is flexible 
> isnofar as it provides a wrapper around any implementation and is to 
> be judged by behaviour.

[bc] The specification does state specific interfaces that are to be 
implemented and how they are to behave. I don't believe this is that 
gray of an area. These interfaces include methods for creating, querying 
and modifying the DOM .

[snip]

>>
> [dd] Where identifiable, I also believe we should try to state that it 
> is the framework, not the implementation, that causes problems. 
> Perhaps we should try to work a bit on the harness itself, then. I 
> suppose similar views could be raised about the JUnit framework.

[bc] For the most part, any exception thrown outside the script 
implementing the specific test of the DOM, in particular any exception 
thrown in the test harness code, can be treated as a failure in the test 
harness and can be reported as such. Inside of the script implementing a 
specific test care must be made to distinguish exceptions thrown by DOM 
interfaces rather than the test script itself.

[snip]

>
>> Concerns regarding use of external Documents
>> ====
>>
>> By creating test Documents through the use of external documents and 
>> the Parser, failures or limitations in the implementation of the 
>> Parser are inappropriately attributed to the implementation of the 
>> DOM Specification being tested.
>>
> [dd] These limitations only occur at build time, not at run time. We 
> could investigate to see if the software we've used produces errors 
> that are later attributed to the implementations tested by the DOM TS. 

[bc] The issues I am raising occur during run time. Documents are loaded 
via Document.load which results in the browser parsing the external 
document thus introducing dependencies on the browser's parser into the 
tests.

>
[snip]

>> Concerns regarding Browser/Vendor specific workarounds
>> ====
>>
>> The current DOM TS provides work arounds for specific implementations 
>> which do not adequately implement the DOM Specification.  These 
>> workarounds allow the implementation to 'appear' to pass the 
>> conformance tests even though in actuality they do not.
>>
>> Example - document.implementation
>> ----
>> Internet Explorer does not implement the document.implementation 
>> interface of the DOM Core Level 1 specification however the DOM TS 
>> provides a work around which creates test documents via a specific 
>> version of the  MSXML ActiveX control.
>>
>> Example - DOMException
>> ----
>> Internet Explorer/MSXML do not implement the DOMException interface 
>> however the DOM TS provides a mapping from Internet Explorer/MSXML's 
>> proprietary exceptions into the DOMException specification and hides 
>> Internet Explorer's lack of conformance.
>>
>> While providing a 'compatible' or 'equivalence' mode for tests for an 
>> implementation in order to provide as much information to the 
>> implementation's developers as well as users of the implementation is 
>> appropriate and should be included in the DOM TS, any such 
>> workarounds should be clearly delineated from the actual Conformance 
>> Tests and be clearly labeled so as to not mislead users of the 
>> Conformance tests as to what is or is not a Standard.
>>
>> All implementations should be judged equally with regard to 
>> conformance and not have the issues involved confused by such 
>> workarounds.
>>
> [dd] I indicated that there can be grey zones above. However, where we 
> identify such levels of difference from the DOM Specification, it 
> should certainly be discussed. 

[bc] Again, I don't consider this to be that gray of an area.

>
>
>> Concerns over the Coverage of the Specification by the DOM TS
>> ====
>>
>> While investigating the code contained in the ECMAScript file 
>> DOMTestCase.js, it became clear that the method 
>> Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did not 
>> test the reported exception but rather always returned true. The same 
>> was true for the ASVDocumentBuilder as well. As already mentioned the 
>> MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test the 
>> actual reported exception but instead mapped the proprietary 
>> exception onto the DOMException code values.
>>
>> This has lead to the question "How much of the DOM Core Level 1 
>> Specification is actually tested by the DOM TS?"
>>
>> The DOM TS should completely cover the DOM Specification being tested 
>> and all test cases should adequately test every implementation for 
>> the DOM TS to have any real meaning.
>>
> [dd] Again, the group's intention was to provide as many tests as 
> possible in order to achieve maximum coverage. For various reasons, 
> mainly since no tests except for the initial NIST load were committed, 
> we had to draw a line somewhere. Since we do not claim to have maximum 
> coverage, however, future releases of the DOM TS will, especially if 
> more companies commit tests, have better coverage. 

[bc] I understand that more tests are needed. My point with these 
examples is that these particular  existing tests are flawed. They are 
defined to return true for any exception thrown for Mozilla which does 
not constitute a test at all and hide the proprietary implementation of 
IE's exceptions. This is not just a question of coverage, but supposed 
coverage which is plainly incorrect.

>
[snip]

>>
> [dd] I agree with most of your points and think that they should be 
> considered for implementation in the next versions of the DOM TS. 
> However, it was anticipated quite early that we would not have a 
> complete test suite, which is indicated in the DOM TS Process document 
> that received acceptance from the DOM WG, IG and other related parties 
> before being published:
>
> <quote>
> (point 2) The DOM test suites are intended to be used as a tool to aid 
> implementors in developing DOM implementations. Validation and 
> certification of these implementations are outside the scope of this 
> work. The tests and test suites will be provided for information and 
> help only. However, we intend to produce as comprehensive, functional 
> and general tests as possible, and this should be the overall goal in 
> designing and implementing the DOM TS.
> </quote>
>
> (http://www.w3.org/2002/01/DOMConformanceTS-
> Process-20020115#requirements)
>
> There is a clear intention to produce as comprehensive, functional and 
> general tests as possible, but to claim is made that these will be 
> complete. 

[bc] Ok, I understand the intent however since these tests do not 
compose a conformance test, that fact should be made very clear so that 
no confusion can arise from the inadvertant appearance of  "Conformance" 
in the documentation.

Received on Thursday, 21 February 2002 20:28:13 UTC