- From: Robert Clary <bclary@netscape.com>
- Date: Thu, 21 Feb 2002 22:39:33 -0500
- To: Mary Brady <mbrady@nist.gov>
- CC: Dimitris Dimitriadis <dimitris@ontologicon.com>, www-dom-ts@w3.org
[bc] Comments in line. Mary Brady wrote: >Bob, > >Thanks for the input. Let me join Dimitris in welcoming you to this effort. >We welcome >any help, but in particular, are very happy to have someone who can help >ensure that >the tests interface well to Mozilla. > >I can see that some of your comments bring up the issue of "conformance". >You are >correct in pointing out that some folks think of the whole nine yards when >the term >"conformance" is used -- that is, a comprehensive test suite, certification, >validation, etc. >I think that you can see that this effort hopes to someday be comprehensive, >but we are >not yet there. > >Additional comments inlined. > >--Mary > [bc] Yes, people do have expectations when "conformance" is used to describe a test. It is these expectations that I wish to make sure are appropriate. I was very disappointed in the low volume of posts when the DOM TS was announced on Slashdot, but as you can see from the comments (the few that there are) <http://slashdot.org/comments.pl?sid=27983&cid=0&pid=0&startat=&threshold=0&mode=nested&commentsort=0&op=Change> these tests are already being used to denigrate Mozilla's implementation of the DOM Core. mozilla.org's contributers consider standard conformance to be one of their highest goals and they work very hard to achieve it. In a fair test, I believe Mozilla's implementation of the DOM Core is superior to Internet Explorer/MSXML's. However I am afraid that people will never know that unless improvements to the DOM TS are made. As an evangelist, it makes my job very difficult when web developers consider the standard to be whatever Internet Explorer does. For the DOM TS to further that view is beyond understanding. And for some background, I did not get involved in this because of my employment with Netscape. I became involved in testing DOM implementations (perhaps a better phrase than testing conformance) before I became an employee of Netscape. I became an employee of Netscape because of my interest in promoting standards based web development. In fact, if you read your email from the Summer of 2000, you will find correspondence with me on this subject. additional comments in line > >----- Original Message ----- >From: "Robert Clary" <bclary@netscape.com> >To: "Dimitris Dimitriadis" <dimitris@ontologicon.com> >Cc: <www-dom-ts@w3.org> >Sent: Thursday, February 21, 2002 5:23 PM >Subject: Re: Concerns regarding the W3 Document Object Model (DOM) >Conformance Test Suites > > >>Dimitris, >> >>Thank you for the prompt reply. >> >>I will try to help with regard to the jsUnit test frame work and other >>issues related to scripting as much as possible. >> >>Hopefully as time progresses the coverage of the test suites will >>increase as more contributions are made. >> >>I understand some of the tradeoffs that have been made in developing the >>tests however these tests can easily be misinterpreted which has lead to >>incorrect statements regarding Mozilla's support for the W3C DOM. >> Whatever the limitations that you have been working under it is >>incorrect to allow an incomplete test suite which itself does not claim >>to test conformance to be represented as such. >> >>Additional comments inline >> >>Dimitris Dimitriadis wrote: >> >>[snip] >> >>>[dd] On rereading, what the second claim above amounts to is that the >>>implementation, if it passes, passes the _test suite_ (a particular >>>version of it, to be exact). So, no conformance with the DOM >>>specification itself is warranted by passing the test suite. >>> >>[bc] I can see your point with regard to the quoted text however this is >>a very fine distinction to make especially where the title of >>http://www.w3.org/DOM/Test/ is "Document Object Model (DOM) Conformance >>Test Suites". Perhaps these should be renamed to "Document Object Model >>(DOM) Test Suites" to remove the possibilitiy of misunderstanding. >> >>>[dd] Also correct, in part. Here we enter a gray zone, as the DOM >>>Specification only dictated what should be done, and not how. So, in >>>those cases where it is obvious that the DOM specification is not >>>followed, you're right. However, the DOM specification is flexible >>>isnofar as it provides a wrapper around any implementation and is to >>>be judged by behaviour. >>> >>[bc] The specification does state specific interfaces that are to be >>implemented and how they are to behave. I don't believe this is that >>gray of an area. These interfaces include methods for creating, querying >>and modifying the DOM . >> >>[snip] >> >>>[dd] Where identifiable, I also believe we should try to state that it >>>is the framework, not the implementation, that causes problems. >>>Perhaps we should try to work a bit on the harness itself, then. I >>>suppose similar views could be raised about the JUnit framework. >>> >>[bc] For the most part, any exception thrown outside the script >>implementing the specific test of the DOM, in particular any exception >>thrown in the test harness code, can be treated as a failure in the test >>harness and can be reported as such. Inside of the script implementing a >>specific test care must be made to distinguish exceptions thrown by DOM >>interfaces rather than the test script itself. >> >>[snip] >> >>>>Concerns regarding use of external Documents >>>>==== >>>> >>>>By creating test Documents through the use of external documents and >>>>the Parser, failures or limitations in the implementation of the >>>>Parser are inappropriately attributed to the implementation of the >>>>DOM Specification being tested. >>>> >>>[dd] These limitations only occur at build time, not at run time. We >>>could investigate to see if the software we've used produces errors >>>that are later attributed to the implementations tested by the DOM TS. >>> >>[bc] The issues I am raising occur during run time. Documents are loaded >>via Document.load which results in the browser parsing the external >>document thus introducing dependencies on the browser's parser into the >>tests. >> >[mb] Yes, this is a limitation of the test suite. As Dimitris pointed out, >we >have raised this issue on several occasions. In fact, in correspondence >that >we've had with Netscape, we though we had it figured out. Obviously, we >still have more work to do... To date, all of the NIST tests >work this way -- it was a design decision early on in our testing effort -- >the >other approach would be to use the DOM to create a DOM tree and then >manipulate it per a particular test. This approach requires that methods >exist >for creating a DOM tree, and that those methods are correctly implemented. >We can't always depend on that to be the case, so we opted for what we >believed at the time to be the lesser of two evils. This approach may >inadvertently >introduce dependencies on a browser's parser -- in fact, we are aware of >some, >such as white-space, and entity expansion issues. Please raise other issues >that >you may see, so that we may zero in on potential solutions. It will take a >while to >sort out creative ways of dealing with the differences, but I think that >they can all >be overcome. This approach has worked well for many java based >implementations. >We are still trying to gain experience with EMCAScript based >implementations, so >any insight that you can provide on how a particular implementation works >would >be helpful. > [bc] One particular area that could be improved is the use of an internal subset of the DTD rather than an external DTD. Mozilla uses a Non-Validating parser and Non-Validating parsers are not required to read an external DTD. If they are not required to read an external DTD how can they be required to supply default attribute values that are specified in an external DTD? > >>[snip] >> >>>>Concerns regarding Browser/Vendor specific workarounds >>>>==== >>>> >>>>The current DOM TS provides work arounds for specific implementations >>>>which do not adequately implement the DOM Specification. These >>>>workarounds allow the implementation to 'appear' to pass the >>>>conformance tests even though in actuality they do not. >>>> >>>>Example - document.implementation >>>>---- >>>>Internet Explorer does not implement the document.implementation >>>>interface of the DOM Core Level 1 specification however the DOM TS >>>>provides a work around which creates test documents via a specific >>>>version of the MSXML ActiveX control. >>>> >>>>Example - DOMException >>>>---- >>>>Internet Explorer/MSXML do not implement the DOMException interface >>>>however the DOM TS provides a mapping from Internet Explorer/MSXML's >>>>proprietary exceptions into the DOMException specification and hides >>>>Internet Explorer's lack of conformance. >>>> >>>>While providing a 'compatible' or 'equivalence' mode for tests for an >>>>implementation in order to provide as much information to the >>>>implementation's developers as well as users of the implementation is >>>>appropriate and should be included in the DOM TS, any such >>>>workarounds should be clearly delineated from the actual Conformance >>>>Tests and be clearly labeled so as to not mislead users of the >>>>Conformance tests as to what is or is not a Standard. >>>> >>>>All implementations should be judged equally with regard to >>>>conformance and not have the issues involved confused by such >>>>workarounds. >>>> >>>[dd] I indicated that there can be grey zones above. However, where we >>>identify such levels of difference from the DOM Specification, it >>>should certainly be discussed. >>> >>[bc] Again, I don't consider this to be that gray of an area. >> > >[mb] Why don't you think this is a gray area. As Jason points out, the spec >reads: > ><quote> >Some languages and object systems do not support the concept of >exceptions. For such systems, error conditions may be indicated using >native error reporting mechanisms. For some bindings, for example, >methods may return error codes similar to those listed in the >corresponding method descriptions. ></quote> > [bc] COM may not allow exceptions however the mapping of HRESULTS to Exceptions in ECMAScript is possible, and is the basis for the cross language support of COM and is accomplished in Internet Explorer/MSXML.. > >How does Mozilla deal with exceptions? > [bc] I have asked one of our developers to address this in more detail. But let me say that Mozilla is based upon xpCOM which is based upon the same principles as COM. xpCOM does not allow exceptions any more than COM does. However, mozilla.org contributers made certain to throw a DOMException which included the standard code attribute. See http://lxr.mozilla.org/seamonkey/source/dom/public/idl/core/nsIDOMDOMException.idl#58 for the interface definition of DOM Exceptions in Mozilla. Non standard DOMExceptions only make the life of web developers more difficult. If you were a web developer and had to choose between the W3 DOMException Interface which only Mozilla implements or the proprietary one Internet Explorer implements, which would you choose? > >>> >>>>Concerns over the Coverage of the Specification by the DOM TS >>>>==== >>>> >>>>While investigating the code contained in the ECMAScript file >>>>DOMTestCase.js, it became clear that the method >>>>Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did not >>>>test the reported exception but rather always returned true. The same >>>>was true for the ASVDocumentBuilder as well. As already mentioned the >>>>MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test the >>>>actual reported exception but instead mapped the proprietary >>>>exception onto the DOMException code values. >>>> >>>>This has lead to the question "How much of the DOM Core Level 1 >>>>Specification is actually tested by the DOM TS?" >>>> >>>>The DOM TS should completely cover the DOM Specification being tested >>>>and all test cases should adequately test every implementation for >>>>the DOM TS to have any real meaning. >>>> >>>[dd] Again, the group's intention was to provide as many tests as >>>possible in order to achieve maximum coverage. For various reasons, >>>mainly since no tests except for the initial NIST load were committed, >>>we had to draw a line somewhere. Since we do not claim to have maximum >>>coverage, however, future releases of the DOM TS will, especially if >>>more companies commit tests, have better coverage. >>> >>[bc] I understand that more tests are needed. My point with these >>examples is that these particular existing tests are flawed. They are >>defined to return true for any exception thrown for Mozilla which does >>not constitute a test at all and hide the proprietary implementation of >>IE's exceptions. This is not just a question of coverage, but supposed >>coverage which is plainly incorrect. >> >[mb] Please provide a specific example of what is returned by Mozilla. > Mozilla throws a DOMException as described above with (hopefully) the correct value in the code attribute. > >>[snip] >> >>>[dd] I agree with most of your points and think that they should be >>>considered for implementation in the next versions of the DOM TS. >>>However, it was anticipated quite early that we would not have a >>>complete test suite, which is indicated in the DOM TS Process document >>>that received acceptance from the DOM WG, IG and other related parties >>>before being published: >>> >>><quote> >>>(point 2) The DOM test suites are intended to be used as a tool to aid >>>implementors in developing DOM implementations. Validation and >>>certification of these implementations are outside the scope of this >>>work. The tests and test suites will be provided for information and >>>help only. However, we intend to produce as comprehensive, functional >>>and general tests as possible, and this should be the overall goal in >>>designing and implementing the DOM TS. >>></quote> >>> >>>(http://www.w3.org/2002/01/DOMConformanceTS- >>>Process-20020115#requirements) >>> >>>There is a clear intention to produce as comprehensive, functional and >>>general tests as possible, but to claim is made that these will be >>>complete. >>> >>[bc] Ok, I understand the intent however since these tests do not >>compose a conformance test, that fact should be made very clear so that >>no confusion can arise from the inadvertant appearance of "Conformance" >>in the documentation. >> >> >[mb] I'm not sure I follow you here. With respect to the term >"Conformance", I >think that the web pages for the test suite are clear in what is meant. I >do understand >that "conformance" can imply different things to different folks -- that's >why so much >effort was expended in defining what the test suite process is all about. I >would hope >that folks aren't reaching conclusions without first reading this >information. As far as >the tests themselves go, each test was created because there was a clause in >the specification >that defined what should happen. We tried to ensure that each test was >well-grounded >in the specification, and in the xml representation of the test, there are >subject pointers to >the area of the spec being tested. In addition, here at NIST, we have the >actual clauses >from the spec that were used to develop the tests. If you believe a test to >be flawed, >please identify a specific test, and discuss why you believe it to not be >grounded in >the specification. These tests represent a first pass -- through an open >and iterative process, >we hope to improve implementations, the specification, and of course the >tests themselves. > [bc] So long as their is no confusion in the web developer community I would agree with you. However, if there is confusion as to the meaning of the results of the DOM TS, then I think we need to address that fact. Let me state once and for all. It is not my mission to fudge/nudge the DOM TS so that Mozilla performs well on the DOM TS. It is my mission to make the DOM TS as complete, accurate and fair as possible. Whether the DOM TS that results from this effort shows Mozilla's DOM implementation to be adequate or not is secondary. My goal is to provide the developer of any browser that chooses to develop a DOM implementation the means to test it as well as provide to any web developer a fair and accurate way to test the implementations of browsers they wish to support. Such a DOM TS would be a great boon to mozilla.org developers and would help them pinpoint errors in their implementation. You can be assured that bugs would be filed in our database immediately. In fact, I became acutely aware of the DOM TS when mozilla.org contributers began filing bugs against the browser based upon the DOM TS results. It was then I discovered how inadequately Mozilla was being tested. Mary, Thanks for replying. I look forward to working with all of you. Bob
Received on Thursday, 21 February 2002 22:44:37 UTC