Re: Concerns regarding the W3 Document Object Model (DOM) Conformance Test Suites

[bc] Comments in line.

Mary Brady wrote:

>Bob,
>
>Thanks for the input.  Let me join Dimitris in welcoming you to this effort.
>We welcome
>any help, but in particular, are very happy to have someone who can help
>ensure that
>the tests interface well to Mozilla.
>
>I can see that some of your comments bring up the issue of "conformance".
>You are
>correct in pointing out that some folks think of the whole nine yards when
>the term
>"conformance" is used -- that is, a comprehensive test suite, certification,
>validation, etc.
>I think that you can see that this effort hopes to someday be comprehensive,
>but we are
>not yet there.
>
>Additional comments inlined.
>
>--Mary
>
[bc]

Yes, people do have expectations when "conformance" is used to describe 
a test. It is these expectations that I wish to make sure are appropriate.

I was very disappointed in the low volume of posts when the DOM TS was 
announced on Slashdot, but as you can see from the comments (the few 
that there are)

 <http://slashdot.org/comments.pl?sid=27983&cid=0&pid=0&startat=&threshold=0&mode=nested&commentsort=0&op=Change>

these tests are already being used to denigrate Mozilla's implementation 
of the DOM Core.

mozilla.org's contributers consider standard conformance to be one of 
their highest goals and they work very hard to achieve it.  In a fair 
test, I believe Mozilla's implementation of the DOM Core is superior to 
Internet Explorer/MSXML's. However I am afraid that people will never 
know that unless improvements to the DOM TS are made.

As an evangelist,  it makes my job very difficult when web developers 
consider the standard to be whatever Internet Explorer does. For the DOM 
TS to further that view is beyond understanding.

And for some background, I did not get involved in this because of my 
employment with Netscape. I became involved in testing DOM 
implementations (perhaps a better phrase than testing conformance) 
before I became an employee of Netscape. I became an employee of 
Netscape because of my interest in promoting standards based web 
development. In fact, if you read your email from the Summer of 2000, 
you will find correspondence with me on this subject.

additional comments in line

>
>----- Original Message -----
>From: "Robert Clary" <bclary@netscape.com>
>To: "Dimitris Dimitriadis" <dimitris@ontologicon.com>
>Cc: <www-dom-ts@w3.org>
>Sent: Thursday, February 21, 2002 5:23 PM
>Subject: Re: Concerns regarding the W3 Document Object Model (DOM)
>Conformance Test Suites
>
>
>>Dimitris,
>>
>>Thank you for the prompt reply.
>>
>>I will try to help with regard to the jsUnit test frame work and other
>>issues related to scripting as much as possible.
>>
>>Hopefully as time progresses the coverage of the test suites will
>>increase as more contributions are made.
>>
>>I understand some of the tradeoffs that have been made in developing the
>>tests however these tests can easily be misinterpreted which has lead to
>>incorrect statements regarding Mozilla's support for the W3C DOM.
>> Whatever the limitations that you have been working under it is
>>incorrect to allow an incomplete test suite which itself does not claim
>>to test conformance to be represented as such.
>>
>>Additional comments inline
>>
>>Dimitris Dimitriadis wrote:
>>
>>[snip]
>>
>>>[dd] On rereading, what the second claim above amounts to is that the
>>>implementation, if it passes, passes the _test suite_ (a particular
>>>version of it, to be exact). So, no conformance with the DOM
>>>specification itself is warranted by passing the test suite.
>>>
>>[bc] I can see your point with regard to the quoted text however this is
>>a very fine distinction to make especially  where the title of
>>http://www.w3.org/DOM/Test/ is "Document Object Model (DOM) Conformance
>>Test Suites". Perhaps these should be renamed to "Document Object Model
>>(DOM)  Test Suites" to remove the possibilitiy of misunderstanding.
>>
>>>[dd] Also correct, in part. Here we enter a gray zone, as the DOM
>>>Specification only dictated what should be done, and not how. So, in
>>>those cases where it is obvious that the DOM specification is not
>>>followed, you're right. However, the DOM specification is flexible
>>>isnofar as it provides a wrapper around any implementation and is to
>>>be judged by behaviour.
>>>
>>[bc] The specification does state specific interfaces that are to be
>>implemented and how they are to behave. I don't believe this is that
>>gray of an area. These interfaces include methods for creating, querying
>>and modifying the DOM .
>>
>>[snip]
>>
>>>[dd] Where identifiable, I also believe we should try to state that it
>>>is the framework, not the implementation, that causes problems.
>>>Perhaps we should try to work a bit on the harness itself, then. I
>>>suppose similar views could be raised about the JUnit framework.
>>>
>>[bc] For the most part, any exception thrown outside the script
>>implementing the specific test of the DOM, in particular any exception
>>thrown in the test harness code, can be treated as a failure in the test
>>harness and can be reported as such. Inside of the script implementing a
>>specific test care must be made to distinguish exceptions thrown by DOM
>>interfaces rather than the test script itself.
>>
>>[snip]
>>
>>>>Concerns regarding use of external Documents
>>>>====
>>>>
>>>>By creating test Documents through the use of external documents and
>>>>the Parser, failures or limitations in the implementation of the
>>>>Parser are inappropriately attributed to the implementation of the
>>>>DOM Specification being tested.
>>>>
>>>[dd] These limitations only occur at build time, not at run time. We
>>>could investigate to see if the software we've used produces errors
>>>that are later attributed to the implementations tested by the DOM TS.
>>>
>>[bc] The issues I am raising occur during run time. Documents are loaded
>>via Document.load which results in the browser parsing the external
>>document thus introducing dependencies on the browser's parser into the
>>tests.
>>
>[mb] Yes, this is a limitation of the test suite.  As Dimitris pointed out,
>we
>have raised this issue on several occasions.  In fact, in correspondence
>that
>we've had with Netscape, we though we had it figured out.  Obviously, we
>still have more work to do...          To date, all of the NIST tests
>work this way -- it was a design decision early on in our testing effort --
>the
>other approach would be to use the DOM to create a DOM tree and then
>manipulate it per a particular test.  This approach requires that methods
>exist
>for creating a DOM tree, and that those methods are correctly implemented.
>We can't always depend on that to be the case, so we opted for what we
>believed at the time to be the lesser of two evils.  This approach may
>inadvertently
>introduce dependencies on a browser's parser -- in fact, we are aware of
>some,
>such as white-space, and entity expansion issues.  Please raise other issues
>that
>you may see, so that we may zero in on potential solutions.  It will take a
>while to
>sort out creative ways of dealing with the differences, but I think that
>they can all
>be overcome.  This approach has worked well for many java based
>implementations.
>We are still trying to gain experience with EMCAScript based
>implementations, so
>any insight that you can provide on how a particular implementation works
>would
>be helpful.
>
[bc]
One particular area that could be improved is the use of an internal 
subset of the DTD rather than an external DTD. Mozilla uses a 
Non-Validating parser and Non-Validating parsers are not required to 
read an external DTD.  If they are not required to read an external  DTD 
how can they be required to supply default attribute values that are 
specified in an external DTD?

>
>>[snip]
>>
>>>>Concerns regarding Browser/Vendor specific workarounds
>>>>====
>>>>
>>>>The current DOM TS provides work arounds for specific implementations
>>>>which do not adequately implement the DOM Specification.  These
>>>>workarounds allow the implementation to 'appear' to pass the
>>>>conformance tests even though in actuality they do not.
>>>>
>>>>Example - document.implementation
>>>>----
>>>>Internet Explorer does not implement the document.implementation
>>>>interface of the DOM Core Level 1 specification however the DOM TS
>>>>provides a work around which creates test documents via a specific
>>>>version of the  MSXML ActiveX control.
>>>>
>>>>Example - DOMException
>>>>----
>>>>Internet Explorer/MSXML do not implement the DOMException interface
>>>>however the DOM TS provides a mapping from Internet Explorer/MSXML's
>>>>proprietary exceptions into the DOMException specification and hides
>>>>Internet Explorer's lack of conformance.
>>>>
>>>>While providing a 'compatible' or 'equivalence' mode for tests for an
>>>>implementation in order to provide as much information to the
>>>>implementation's developers as well as users of the implementation is
>>>>appropriate and should be included in the DOM TS, any such
>>>>workarounds should be clearly delineated from the actual Conformance
>>>>Tests and be clearly labeled so as to not mislead users of the
>>>>Conformance tests as to what is or is not a Standard.
>>>>
>>>>All implementations should be judged equally with regard to
>>>>conformance and not have the issues involved confused by such
>>>>workarounds.
>>>>
>>>[dd] I indicated that there can be grey zones above. However, where we
>>>identify such levels of difference from the DOM Specification, it
>>>should certainly be discussed.
>>>
>>[bc] Again, I don't consider this to be that gray of an area.
>>
>
>[mb] Why don't you think this is a gray area.  As Jason points out, the spec
>reads:
>
><quote>
>Some languages and object systems do not support the concept of
>exceptions. For such systems, error conditions may be indicated using
>native error reporting mechanisms. For some bindings, for example,
>methods may return error codes similar to those listed in the
>corresponding method descriptions.
></quote>
>
[bc]

COM may not allow exceptions however the mapping of HRESULTS to 
Exceptions in ECMAScript is possible, and is the basis for the cross 
language support of COM and is accomplished in Internet Explorer/MSXML..  

>
>How does Mozilla deal with exceptions?
>
[bc]

I have asked one of our developers to address this in more detail.

But let me say that Mozilla is based upon xpCOM which is based upon the 
same principles as COM.

xpCOM does not allow exceptions any more than COM does. However, 
mozilla.org contributers made certain to throw a DOMException which 
included the standard code attribute.

See 
http://lxr.mozilla.org/seamonkey/source/dom/public/idl/core/nsIDOMDOMException.idl#58 
for the interface definition of DOM Exceptions in Mozilla.

Non standard DOMExceptions only make the life of web developers more 
difficult. If you were a web developer and had to choose between the W3 
DOMException Interface which only Mozilla implements or the proprietary 
one Internet Explorer implements, which would you choose? 

>
>>>
>>>>Concerns over the Coverage of the Specification by the DOM TS
>>>>====
>>>>
>>>>While investigating the code contained in the ECMAScript file
>>>>DOMTestCase.js, it became clear that the method
>>>>Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did not
>>>>test the reported exception but rather always returned true. The same
>>>>was true for the ASVDocumentBuilder as well. As already mentioned the
>>>>MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test the
>>>>actual reported exception but instead mapped the proprietary
>>>>exception onto the DOMException code values.
>>>>
>>>>This has lead to the question "How much of the DOM Core Level 1
>>>>Specification is actually tested by the DOM TS?"
>>>>
>>>>The DOM TS should completely cover the DOM Specification being tested
>>>>and all test cases should adequately test every implementation for
>>>>the DOM TS to have any real meaning.
>>>>
>>>[dd] Again, the group's intention was to provide as many tests as
>>>possible in order to achieve maximum coverage. For various reasons,
>>>mainly since no tests except for the initial NIST load were committed,
>>>we had to draw a line somewhere. Since we do not claim to have maximum
>>>coverage, however, future releases of the DOM TS will, especially if
>>>more companies commit tests, have better coverage.
>>>
>>[bc] I understand that more tests are needed. My point with these
>>examples is that these particular  existing tests are flawed. They are
>>defined to return true for any exception thrown for Mozilla which does
>>not constitute a test at all and hide the proprietary implementation of
>>IE's exceptions. This is not just a question of coverage, but supposed
>>coverage which is plainly incorrect.
>>
>[mb] Please provide a specific example of what is returned by Mozilla.
>
Mozilla throws a DOMException as described above with (hopefully) the 
correct value in the code attribute.

>
>>[snip]
>>
>>>[dd] I agree with most of your points and think that they should be
>>>considered for implementation in the next versions of the DOM TS.
>>>However, it was anticipated quite early that we would not have a
>>>complete test suite, which is indicated in the DOM TS Process document
>>>that received acceptance from the DOM WG, IG and other related parties
>>>before being published:
>>>
>>><quote>
>>>(point 2) The DOM test suites are intended to be used as a tool to aid
>>>implementors in developing DOM implementations. Validation and
>>>certification of these implementations are outside the scope of this
>>>work. The tests and test suites will be provided for information and
>>>help only. However, we intend to produce as comprehensive, functional
>>>and general tests as possible, and this should be the overall goal in
>>>designing and implementing the DOM TS.
>>></quote>
>>>
>>>(http://www.w3.org/2002/01/DOMConformanceTS-
>>>Process-20020115#requirements)
>>>
>>>There is a clear intention to produce as comprehensive, functional and
>>>general tests as possible, but to claim is made that these will be
>>>complete.
>>>
>>[bc] Ok, I understand the intent however since these tests do not
>>compose a conformance test, that fact should be made very clear so that
>>no confusion can arise from the inadvertant appearance of  "Conformance"
>>in the documentation.
>>
>>
>[mb] I'm not sure I follow you here.  With respect to the term
>"Conformance", I
>think that the web pages for the test suite are clear in what is meant.  I
>do understand
>that "conformance" can imply different things to different folks -- that's
>why so much
>effort was expended in defining what the test suite process is all about.  I
>would hope
>that folks aren't reaching conclusions without first reading this
>information.  As far as
>the tests themselves go, each test was created because there was a clause in
>the specification
>that defined what should happen.  We tried to ensure that each test was
>well-grounded
>in the specification, and in the xml representation of the test, there are
>subject pointers to
>the area of the spec being tested.  In addition, here at NIST, we have the
>actual clauses
>from the spec that were used to develop the tests.  If you believe a test to
>be flawed,
>please identify a specific test, and discuss why you believe it to not be
>grounded in
>the specification.  These tests represent a first pass -- through an open
>and iterative process,
>we hope to improve implementations, the specification, and of course the
>tests themselves.
>
[bc]

So long as their is no confusion in  the web developer community I would 
agree with you. However, if there is confusion as to the meaning of the 
results of the DOM TS, then I think we need to address that  fact.

Let me state once and for all. It is not my mission to fudge/nudge the 
DOM TS so that Mozilla performs well on the DOM TS. It is my mission to 
make the DOM TS as complete, accurate and fair as possible. Whether the 
DOM TS that results from this effort shows Mozilla's DOM implementation 
to be adequate or not is secondary.  My goal is to provide the developer 
of any browser that chooses to develop a  DOM implementation the means 
to test it as well as provide to any web developer a fair and accurate 
way to test the implementations of browsers they wish to support.

Such a DOM TS would be a great boon to mozilla.org developers and would 
help them pinpoint errors in their implementation. You can be assured 
that bugs would be filed in our database immediately. In fact, I became 
acutely aware of the DOM TS when mozilla.org contributers began filing 
bugs against the browser based upon the DOM TS results. It was then I 
discovered how inadequately Mozilla was being tested. 

Mary, Thanks for replying. I look forward to working with all of you.

Bob

Received on Thursday, 21 February 2002 22:44:37 UTC