Re: Concerns regarding the W3 Document Object Model (DOM) Conformance Test Suites

Bob,

Thanks for the input.  Let me join Dimitris in welcoming you to this effort.
We welcome
any help, but in particular, are very happy to have someone who can help
ensure that
the tests interface well to Mozilla.

I can see that some of your comments bring up the issue of "conformance".
You are
correct in pointing out that some folks think of the whole nine yards when
the term
"conformance" is used -- that is, a comprehensive test suite, certification,
validation, etc.
I think that you can see that this effort hopes to someday be comprehensive,
but we are
not yet there.

Additional comments inlined.

--Mary

----- Original Message -----
From: "Robert Clary" <bclary@netscape.com>
To: "Dimitris Dimitriadis" <dimitris@ontologicon.com>
Cc: <www-dom-ts@w3.org>
Sent: Thursday, February 21, 2002 5:23 PM
Subject: Re: Concerns regarding the W3 Document Object Model (DOM)
Conformance Test Suites


> Dimitris,
>
> Thank you for the prompt reply.
>
> I will try to help with regard to the jsUnit test frame work and other
> issues related to scripting as much as possible.
>
> Hopefully as time progresses the coverage of the test suites will
> increase as more contributions are made.
>
> I understand some of the tradeoffs that have been made in developing the
> tests however these tests can easily be misinterpreted which has lead to
> incorrect statements regarding Mozilla's support for the W3C DOM.
>  Whatever the limitations that you have been working under it is
> incorrect to allow an incomplete test suite which itself does not claim
> to test conformance to be represented as such.
>
> Additional comments inline
>
> Dimitris Dimitriadis wrote:
>
> [snip]
>
> >>
> > [dd] On rereading, what the second claim above amounts to is that the
> > implementation, if it passes, passes the _test suite_ (a particular
> > version of it, to be exact). So, no conformance with the DOM
> > specification itself is warranted by passing the test suite.
>
> [bc] I can see your point with regard to the quoted text however this is
> a very fine distinction to make especially  where the title of
> http://www.w3.org/DOM/Test/ is "Document Object Model (DOM) Conformance
> Test Suites". Perhaps these should be renamed to "Document Object Model
> (DOM)  Test Suites" to remove the possibilitiy of misunderstanding.
>
> > [dd] Also correct, in part. Here we enter a gray zone, as the DOM
> > Specification only dictated what should be done, and not how. So, in
> > those cases where it is obvious that the DOM specification is not
> > followed, you're right. However, the DOM specification is flexible
> > isnofar as it provides a wrapper around any implementation and is to
> > be judged by behaviour.
>
> [bc] The specification does state specific interfaces that are to be
> implemented and how they are to behave. I don't believe this is that
> gray of an area. These interfaces include methods for creating, querying
> and modifying the DOM .
>
> [snip]
>
> >>
> > [dd] Where identifiable, I also believe we should try to state that it
> > is the framework, not the implementation, that causes problems.
> > Perhaps we should try to work a bit on the harness itself, then. I
> > suppose similar views could be raised about the JUnit framework.
>
> [bc] For the most part, any exception thrown outside the script
> implementing the specific test of the DOM, in particular any exception
> thrown in the test harness code, can be treated as a failure in the test
> harness and can be reported as such. Inside of the script implementing a
> specific test care must be made to distinguish exceptions thrown by DOM
> interfaces rather than the test script itself.
>
> [snip]
>
> >
> >> Concerns regarding use of external Documents
> >> ====
> >>
> >> By creating test Documents through the use of external documents and
> >> the Parser, failures or limitations in the implementation of the
> >> Parser are inappropriately attributed to the implementation of the
> >> DOM Specification being tested.
> >>
> > [dd] These limitations only occur at build time, not at run time. We
> > could investigate to see if the software we've used produces errors
> > that are later attributed to the implementations tested by the DOM TS.
>
> [bc] The issues I am raising occur during run time. Documents are loaded
> via Document.load which results in the browser parsing the external
> document thus introducing dependencies on the browser's parser into the
> tests.
>
[mb] Yes, this is a limitation of the test suite.  As Dimitris pointed out,
we
have raised this issue on several occasions.  In fact, in correspondence
that
we've had with Netscape, we though we had it figured out.  Obviously, we
still have more work to do...          To date, all of the NIST tests
work this way -- it was a design decision early on in our testing effort --
the
other approach would be to use the DOM to create a DOM tree and then
manipulate it per a particular test.  This approach requires that methods
exist
for creating a DOM tree, and that those methods are correctly implemented.
We can't always depend on that to be the case, so we opted for what we
believed at the time to be the lesser of two evils.  This approach may
inadvertently
introduce dependencies on a browser's parser -- in fact, we are aware of
some,
such as white-space, and entity expansion issues.  Please raise other issues
that
you may see, so that we may zero in on potential solutions.  It will take a
while to
sort out creative ways of dealing with the differences, but I think that
they can all
be overcome.  This approach has worked well for many java based
implementations.
We are still trying to gain experience with EMCAScript based
implementations, so
any insight that you can provide on how a particular implementation works
would
be helpful.

> >
> [snip]
>
> >> Concerns regarding Browser/Vendor specific workarounds
> >> ====
> >>
> >> The current DOM TS provides work arounds for specific implementations
> >> which do not adequately implement the DOM Specification.  These
> >> workarounds allow the implementation to 'appear' to pass the
> >> conformance tests even though in actuality they do not.
> >>
> >> Example - document.implementation
> >> ----
> >> Internet Explorer does not implement the document.implementation
> >> interface of the DOM Core Level 1 specification however the DOM TS
> >> provides a work around which creates test documents via a specific
> >> version of the  MSXML ActiveX control.
> >>
> >> Example - DOMException
> >> ----
> >> Internet Explorer/MSXML do not implement the DOMException interface
> >> however the DOM TS provides a mapping from Internet Explorer/MSXML's
> >> proprietary exceptions into the DOMException specification and hides
> >> Internet Explorer's lack of conformance.
> >>
> >> While providing a 'compatible' or 'equivalence' mode for tests for an
> >> implementation in order to provide as much information to the
> >> implementation's developers as well as users of the implementation is
> >> appropriate and should be included in the DOM TS, any such
> >> workarounds should be clearly delineated from the actual Conformance
> >> Tests and be clearly labeled so as to not mislead users of the
> >> Conformance tests as to what is or is not a Standard.
> >>
> >> All implementations should be judged equally with regard to
> >> conformance and not have the issues involved confused by such
> >> workarounds.
> >>
> > [dd] I indicated that there can be grey zones above. However, where we
> > identify such levels of difference from the DOM Specification, it
> > should certainly be discussed.
>
> [bc] Again, I don't consider this to be that gray of an area.

[mb] Why don't you think this is a gray area.  As Jason points out, the spec
reads:

<quote>
Some languages and object systems do not support the concept of
exceptions. For such systems, error conditions may be indicated using
native error reporting mechanisms. For some bindings, for example,
methods may return error codes similar to those listed in the
corresponding method descriptions.
</quote>

How does Mozilla deal with exceptions?

>
> >
> >
> >> Concerns over the Coverage of the Specification by the DOM TS
> >> ====
> >>
> >> While investigating the code contained in the ECMAScript file
> >> DOMTestCase.js, it became clear that the method
> >> Moz[XML|HTML]DocumentBuilder_isDOMExceptionCode(ex, code) did not
> >> test the reported exception but rather always returned true. The same
> >> was true for the ASVDocumentBuilder as well. As already mentioned the
> >> MS[XML|HTML]DocumentBuilder_isDOMExceptionCode did not test the
> >> actual reported exception but instead mapped the proprietary
> >> exception onto the DOMException code values.
> >>
> >> This has lead to the question "How much of the DOM Core Level 1
> >> Specification is actually tested by the DOM TS?"
> >>
> >> The DOM TS should completely cover the DOM Specification being tested
> >> and all test cases should adequately test every implementation for
> >> the DOM TS to have any real meaning.
> >>
> > [dd] Again, the group's intention was to provide as many tests as
> > possible in order to achieve maximum coverage. For various reasons,
> > mainly since no tests except for the initial NIST load were committed,
> > we had to draw a line somewhere. Since we do not claim to have maximum
> > coverage, however, future releases of the DOM TS will, especially if
> > more companies commit tests, have better coverage.
>
> [bc] I understand that more tests are needed. My point with these
> examples is that these particular  existing tests are flawed. They are
> defined to return true for any exception thrown for Mozilla which does
> not constitute a test at all and hide the proprietary implementation of
> IE's exceptions. This is not just a question of coverage, but supposed
> coverage which is plainly incorrect.
>
[mb] Please provide a specific example of what is returned by Mozilla.

> >
> [snip]
>
> >>
> > [dd] I agree with most of your points and think that they should be
> > considered for implementation in the next versions of the DOM TS.
> > However, it was anticipated quite early that we would not have a
> > complete test suite, which is indicated in the DOM TS Process document
> > that received acceptance from the DOM WG, IG and other related parties
> > before being published:
> >
> > <quote>
> > (point 2) The DOM test suites are intended to be used as a tool to aid
> > implementors in developing DOM implementations. Validation and
> > certification of these implementations are outside the scope of this
> > work. The tests and test suites will be provided for information and
> > help only. However, we intend to produce as comprehensive, functional
> > and general tests as possible, and this should be the overall goal in
> > designing and implementing the DOM TS.
> > </quote>
> >
> > (http://www.w3.org/2002/01/DOMConformanceTS-
> > Process-20020115#requirements)
> >
> > There is a clear intention to produce as comprehensive, functional and
> > general tests as possible, but to claim is made that these will be
> > complete.
>
> [bc] Ok, I understand the intent however since these tests do not
> compose a conformance test, that fact should be made very clear so that
> no confusion can arise from the inadvertant appearance of  "Conformance"
> in the documentation.
>
>
[mb] I'm not sure I follow you here.  With respect to the term
"Conformance", I
think that the web pages for the test suite are clear in what is meant.  I
do understand
that "conformance" can imply different things to different folks -- that's
why so much
effort was expended in defining what the test suite process is all about.  I
would hope
that folks aren't reaching conclusions without first reading this
information.  As far as
the tests themselves go, each test was created because there was a clause in
the specification
that defined what should happen.  We tried to ensure that each test was
well-grounded
in the specification, and in the xml representation of the test, there are
subject pointers to
the area of the spec being tested.  In addition, here at NIST, we have the
actual clauses
from the spec that were used to develop the tests.  If you believe a test to
be flawed,
please identify a specific test, and discuss why you believe it to not be
grounded in
the specification.  These tests represent a first pass -- through an open
and iterative process,
we hope to improve implementations, the specification, and of course the
tests themselves.

>

Received on Thursday, 21 February 2002 21:33:51 UTC