Re: Proposal for ECMAScript binding rewrite

Curt Arnold wrote:
> 
> Sorry about the delay on responding.  I'm unclear on the difference 
> between "Parser Configuration" and "Test Implementation Attributes".  Is 
> the latter is a way of subselecting tests that, for example, require 
> namespace awareness?  Could you explain the motivation for that, I don't 
> see it.
> 

I separated the parser configuration and test selection implementation 
attributes for the following reasons:

1. The ECMAScript binding is fundamentally different from the Java 
binding. For the Java binding, the test framework must be customized for 
each implementation where the default values and potential for 
customization for the parser are well known. For the ECMAScript binding, 
any browser which supports the JsUnit test framework can potentially run 
the tests using the minimal IFRAME implementation however much less is 
known about the parser's default behavior and potential for customization.

2. Tests do not have a full set of implementation attributes included. 
   When missing from a test, some implementation attributes would be 
expected to default to false while others would expect to be defaulted 
to true. Ideally, each test would include a full set of implementation 
features which it expects rather than rely on the interpretation of a 
missing value.

3. The previous method of using one set of implementation attributes to 
both configure the parser and to select tests caused problems in cases 
where the parser's value was assumed to be a specific value and 
unchangeable. For example, IFRAME implementation had "Namespace aware" 
false which was not modifiable. This made it impossible for Mozilla to 
run namespace aware tests.

For these reasons, I separated the Parser Configuration from the Test 
Selection. This separation allows the user to select any combination of 
parameters without being limited in the choices possible. This puts the 
user in full control and makes it it explicitly clear what selection 
criteria are in use and the effects of the choices.

The parser implementation attributes are used solely for configuring the 
parser if that is possible. I introduced a third value for the parser 
implementation attributes for the cases where the actual behavior used 
was the parser's default which was not known.

The test implementation attributes are used solely for selecting tests 
which match the user's selection. I introduced a third value for the 
test implementation attribute which currently allows the user to ignore 
a particular implementation attribute when making test selections.

> Could you also explain what you were trying to do with the test feature 
> and version combo?  If I select "XML" and "Ignore", then 83 tests are 
> identified as compatible and 439 are identified as incompatible, however 
> all L1 Core tests are compatible with an implementation returning true 
> for hasFeature("XML", null).  Maybe only 83 explicitly declare that they 
> require "XML".
> 

The feature selection does in fact select only those tests which 
explicitly set hasFeature. It does not take into account the return 
value of DOMImplementation.hasFeature at all. I believe using the 
DOMImplementation.hasFeature to select tests is incorrect.

If I understand the specification for hasFeature correctly, an 
implementation is required to return false unless it supports the full 
set of fundamental interfaces. Using the return value of hasFeature to 
automatically select tests would not allow an implementation to test a 
partial implementation which returned false for hasFeature.

For the <hasFeature> element of the DOM TS Markup language to be fully 
useful it should be specified explicitly in each test. However, a user 
should be able to choose to run a test regardless of a particular 
implementation's return value for hasFeature or the <hasFeature> setting 
in a test.


> There is a requirement to be able to tweak the parser configuration 
> (when possible).  The list of compatible and incompatible tests was just 
> to provide some feedback since the JSUnit will report incompatible tests 
> as being successfully completed where they won't appear in a JUnit run.  
> If there is a need to be able to find, for example, all tests that 
> require feature "HTML" to be true, then that should probably be in the 
> test matrix since it would apply to all platform bindings.
> 

This approach allows the fullest possible configuration of the parser 
for cases where it is known that the parser is configurable. For 
example, MSXML3 and MSXML4 are fully configurable.

Ideally the  new L3 interfaces could be used for conforming 
implementations to discover the default values and configurable parser 
features however I have not attempted that in this iteration.

The invisible selection of tests based upon unmodifiable decisions made 
in a test removes choice from the user. Allowing an implementation to 
arbitrarily run a test is an important consideration for implementors. 
For example, I discovered a new Mozilla crash bug when Mozilla was 
allowed to run the full set of tests. Since Mozilla might well run into 
this type of code in the "wild" it was a good thing to identify the 
problem and fix it.

> If we could add a "warning" result to JSUnit, then we could skip the 
> listing of compatible and incompatible tests in the test configuration. 
>  The incompatible tests could just note that they were effectively 
> skipped during test evaluation.
> 

I do think that using JsUnit's trace facility would be very useful and 
have considered possibilities for it's use however I do not think that 
it is reasonable for a test to reject being run. That choice should 
remain with the user.

/bc