Re: Proposal for ECMAScript binding rewrite

On Mar 1, 2004, at 3:32 PM, Bob Clary wrote:

> Curt Arnold wrote:
>> Sorry about the delay on responding.  I'm unclear on the difference 
>> between "Parser Configuration" and "Test Implementation Attributes".  
>> Is the latter is a way of subselecting tests that, for example, 
>> require namespace awareness?  Could you explain the motivation for 
>> that, I don't see it.
>
> I separated the parser configuration and test selection implementation 
> attributes for the following reasons:
>
> 1. The ECMAScript binding is fundamentally different from the Java 
> binding. For the Java binding, the test framework must be customized 
> for each implementation where the default values and potential for 
> customization for the parser are well known. For the ECMAScript 
> binding, any browser which supports the JsUnit test framework can 
> potentially run the tests using the minimal IFRAME implementation 
> however much less is known about the parser's default behavior and 
> potential for customization.
>
> 2. Tests do not have a full set of implementation attributes included. 
>   When missing from a test, some implementation attributes would be 
> expected to default to false while others would expect to be defaulted 
> to true. Ideally, each test would include a full set of implementation 
> features which it expects rather than rely on the interpretation of a 
> missing value.


Tests should ideally the smallest number of preconditions possible.  
Each precondition makes the test worthless for some class of conformant 
processor.  If I specify,

<implementationAttribute name="namespaceAware" value="true"/>

That means that namespace awareness is essential to the test.  The 
value of any other implementation property, expanding entity 
references, for example, should not matter.  The default Java runner 
runs the test twice, the second default configuration has every 
property inverted from the first.  Only properties that are asserted to 
a specific value are forced that way.

>
> 3. The previous method of using one set of implementation attributes 
> to both configure the parser and to select tests caused problems in 
> cases where the parser's value was assumed to be a specific value and 
> unchangeable. For example, IFRAME implementation had "Namespace aware" 
> false which was not modifiable. This made it impossible for Mozilla to 
> run namespace aware tests.

I can see where namespace aware should have been true for XHTML using 
the IFrame loader.

>
> For these reasons, I separated the Parser Configuration from the Test 
> Selection. This separation allows the user to select any combination 
> of parameters without being limited in the choices possible. This puts 
> the user in full control and makes it it explicitly clear what 
> selection criteria are in use and the effects of the choices.

> The parser implementation attributes are used solely for configuring 
> the parser if that is possible. I introduced a third value for the 
> parser implementation attributes for the cases where the actual 
> behavior used was the parser's default which was not known.
>
> The test implementation attributes are used solely for selecting tests 
> which match the user's selection. I introduced a third value for the 
> test implementation attribute which currently allows the user to 
> ignore a particular implementation attribute when making test 
> selections.
>
>> Could you also explain what you were trying to do with the test 
>> feature and version combo?  If I select "XML" and "Ignore", then 83 
>> tests are identified as compatible and 439 are identified as 
>> incompatible, however all L1 Core tests are compatible with an 
>> implementation returning true for hasFeature("XML", null).  Maybe 
>> only 83 explicitly declare that they require "XML".
>
> The feature selection does in fact select only those tests which 
> explicitly set hasFeature. It does not take into account the return 
> value of DOMImplementation.hasFeature at all. I believe using the 
> DOMImplementation.hasFeature to select tests is incorrect.
>
> If I understand the specification for hasFeature correctly, an 
> implementation is required to return false unless it supports the full 
> set of fundamental interfaces. Using the return value of hasFeature to 
> automatically select tests would not allow an implementation to test a 
> partial implementation which returned false for hasFeature.

It is only a conformance failure if a test claims a certain feature and 
does not deliver it.  In the L1 and L2 tests, checking that 
hasFeature("XML", null) is true is precondition for tests that use 
createEntityReference or any other DOM method that is not guaranteed to 
be in non-XML implementations.  Maybe a non-XML implementation does 
support createEntityReference, but unless it claims hasFeature("XML") I 
can't rely on the test.


>
> For the <hasFeature> element of the DOM TS Markup language to be fully 
> useful it should be specified explicitly in each test. However, a user 
> should be able to choose to run a test regardless of a particular 
> implementation's return value for hasFeature or the <hasFeature> 
> setting in a test.
>
>

It is a very good thing that only the minimal preconditions are 
specified.  If I have tests that work regardless of a namespace 
support, leaving it unspecified allows me to use the same test to check 
potentially different code paths by manipulating the initial settings.  
If a test does not have <hasFeature feature="XML"/>, then the test 
should be equally valid for HTML and XML.



>> There is a requirement to be able to tweak the parser configuration 
>> (when possible).  The list of compatible and incompatible tests was 
>> just to provide some feedback since the JSUnit will report 
>> incompatible tests as being successfully completed where they won't 
>> appear in a JUnit run.  If there is a need to be able to find, for 
>> example, all tests that require feature "HTML" to be true, then that 
>> should probably be in the test matrix since it would apply to all 
>> platform bindings.
>
> This approach allows the fullest possible configuration of the parser 
> for cases where it is known that the parser is configurable. For 
> example, MSXML3 and MSXML4 are fully configurable.
>
> Ideally the  new L3 interfaces could be used for conforming 
> implementations to discover the default values and configurable parser 
> features however I have not attempted that in this iteration.
>
> The invisible selection of tests based upon unmodifiable decisions 
> made in a test removes choice from the user. Allowing an 
> implementation to arbitrarily run a test is an important consideration 
> for implementors. For example, I discovered a new Mozilla crash bug 
> when Mozilla was allowed to run the full set of tests. Since Mozilla 
> might well run into this type of code in the "wild" it was a good 
> thing to identify the problem and fix it.
>



>> If we could add a "warning" result to JSUnit, then we could skip the 
>> listing of compatible and incompatible tests in the test 
>> configuration.  The incompatible tests could just note that they were 
>> effectively skipped during test evaluation.
>
> I do think that using JsUnit's trace facility would be very useful and 
> have considered possibilities for it's use however I do not think that 
> it is reasonable for a test to reject being run. That choice should 
> remain with the user.


I could see adding an "ignore preconditions" or similar option that 
would allow you to bypass the safeties on a test.  However, the 
preconditions should be there for a reason.  I removed a whole lot of 
unnecessary preconditions in the L3 submission, a lot of the tests 
asserted namespace awareness just to make getElementsByTagNameNS to 
work when they could have used getElementsByTagName.

Received on Monday, 1 March 2004 20:03:40 UTC