Re: ECMA harness

One comment on the JsUnit in our framework is the pointer included in 
alltests.html which points to a folder presupposed by the JsUnit 
application. We may want to look at this in order not to tie people to a 
particular setup once we release the suite.

Also, while running the tests in a particular browser, it reported no 
errors (which would be fantastic); this not being the case however, it 
turns out that Junit iterates over all functions in the test file and 
reports a pass rersult even if the tests are not properly executed. We 
may want to do something to safeguard against this. I'll look into this 
some more and generate a more informative report.

I'll try to take up testing error-raising ignorant implementations with 
the DOM WG this week's telephone conference.

/Dimitris
On Monday, November 19, 2001, at 05:14  PM, Mary Brady wrote:

> We have not yet looked at allowing the new
> jsunit-based harness to discriminately run
> tests -- it currently runs all available tests at
> once and provides a report.  Have you tried
> to run this yet Jason?  I'd be interested in
> your reaction.  We do, however, have the
> ability to insert inside a particular test that it
> will only run with a particular module -- we have
> not yet done this -- but I would think that doing
> so would cover the HTML-only tests.  Maybe
> Curt can comment on how we should use
> this feature, and what the transform does
> with this information. 
>  
> --Mary
>  
>
> ----- Original Message -----
> From: Jason Brittsan
> To: Dimitris Dimitriadis
> Cc: Mary Brady ; www-dom-ts@w3.org
> Sent: Friday, November 16, 2001 3:26 PM
> Subject: RE: ECMA harness
>
> Hi Dimitris,
>
> My comment could take on several meanings, depending on the design of
> the test harness.  Originally, I meant that there should be an option to
> only run the tests that match the capabilities of the client in
> reference to HTML-only implementations and implementations that do not
> support exceptions.  This is consistent with discussions that took place
> early on in the development of this test suite.  I'm not proposing any
> modularization other than that.
>
> Depending on the implementation of the harness, we could also provide
> ways of running only certain portions of the test suite, based on the
> needs of the user.  The current NIST harness provides some basic
> functionality by using a SELECT control to specify the "DOM Category"
> and another SELECT control to specify the "DOM Interface."  The test
> case selection process could be made better.  I've attached a *sample*
> of what this could look like.  (THIS IS ONLY A MOCK-UP!)  In addition,
> running the suite would be automated, or at least take less time for a
> person to run the conformance suite.
>
> Flexibility in test case selection would lead to more useful reporting.
> Users would only get back the information they desire instead of sorting
> through test case areas that don't concern them.  Also, the results
> would be posted on a single HTML page (or XML file?) instead of
> requiring the tester to visit each interface, record the results, and
> move on to the next interface.
>
> -Jason
>
> -----Original Message-----
> From: Dimitris Dimitriadis [mailto:dimitris@ontologicon.com]
> Sent: Friday, November 16, 2001 12:23 AM
> To: Jason Brittsan
> Cc: Mary Brady; www-dom-ts@w3.org
> Subject: Re: ECMA harness
>
> Hi Jason
>
> Please provide a more detailed account of what this would mean; running
> different parts of the test suite? Have different test suites built to
> begin with in accordance with existing browser capabilities? Currently
> we haven't limited the suite in any other way than defined by the DOM
> specification, except for entity exapansion and whitespace preservation
> in parsers.
>
> In order to come to an understanding about the harness thus: how should
> we act in this matter? Should we modularize the test suite in some way?
> We have discussed this in the past, and given the fact that we want to
> release the test suite as soon as possible it seems a good idea to make
> this explicit very soon.
>
> In our previous discussion though, we decided to go for using a
> modularization that stayed as close as possible to the DOM
> specification. On the other hand, we obviously want to be able to run
> the test suite in all major browsers. IE can be tested running the Ecma
> tests with the JSunit framework by running ant dom1-core-gen-jsunit to
> build the appropriate code.
>
> /Dimitris
>
>
> On Monday, November 12, 2001, at 02:32  PM, Jason Brittsan wrote:
>
> > Hi Mary... my apologies for not responding sooner.  Today is my first
> > day back from vacation.
> >
> > Of course, I will be happy to provide any assistance I can with the
> test
> > harness.
> >
> > The test harness should be able to run tests based on the capabilities
> > of the client.  Therefore we need to support this in the harness UI.
> >
> > I believe that flexibility in reporting is our best strategy.
> >
> > -----Original Message-----
> > From: Mary Brady [mailto:mbrady@nist.gov]
> > Sent: Tuesday, October 30, 2001 8:39 AM
> > To: www-dom-ts@w3.org
> > Subject: ECMA harness
> >
> > In building the ECMA harness, I have started with the original harness
> > that was provided from the NIST web site:
> >
> >     http://xw2k.sdct.itl.nist.gov/dom/index.html
> > This harness uses whatever DOM implementation is running on the
> > client side, attempts to run available tests, and reports the results.
> > Each of the tests expect to have access to common xml load routines
> > and common assertion routines.  I expect that we can use the same
> > code that is currently being used by the jsunit harness. The following
> > needs to be done:
> >
> > 1) Integrate current load/assertion routines -- Mary
> > 2) Validate load routines
> >             -- IE (Jason)
> >             -- Mozilla (Do we have a Netscape volunteer?)
> > 3) Validate DOMException codes
> >             -- IE (Jason)
> >             -- Mozilla ?
> > 4) Determine high level interface -- all
> >             -- Do we want to run all tests, or be able to
> >                 discriminately pick appropriate tests?
> > 5) Determine reporting mechanism
> >             -- simply dump returns from tests?
> >             -- color-code results?
> >             -- display expected vs actual?
> >             -- possibly modify code to accomodate
> >                 what we want to display.
> > 6) Access to other testing resources ?
> >             -- test assertions, <subjects>
> >             -- view source code
> >             -- view portion of spec being tested.
> >
> > Anything else?
> >
> > --Mary
> >
>