Re: testing plans for Web IDL

Hi Dom.

Dominique Hazael-Massieux:
> I read with great pleasure that there is now a well-defined schedule for
> moving Web IDL to LC: good job!
> http://www.w3.org/mid/4DDC2483.6030401@nokia.com

Let’s get there first. ;)

> Has there been any thoughts given around the CR phrase for Web IDL?
> Given how important that specification is, I'm hoping there will be a
> good test suite to back it up.
> 
> Are there plans to build such a test suite?

I am as yet unsure how we should write the CR exit criteria.  The
specification has a number of conformance classes, the two main ones
being conforming IDL fragments and conforming implementations (for a
particular language).

We want to know if the specification’s requirements are appropriate for
authors of other web platform specifications.  We can see that there are
a number of specifications that do use Web IDL, so that is some evidence
that Web IDL is doing the right thing.  That there are more than n
specifications using Web IDL might be one criterion for progressing past
CR.  Should we also wait to see if there are more than n specifications
using each feature from Web IDL?  Some features are peculiar things that
we don’t necessarily want more specifications using, and which we have
only for the benefit of compatibility in HTML5.  So I am not sure if
“≥ n specs using feature X” is a good exit criterion.

Whether these other specifications write correct IDL fragments is
another thing to look at, and something we can (and you do) test for.
This corresponds to the “conforming IDL fragment” conformance class.
If specification writers are writing IDL fragments correctly, and
haven’t asked for the changes to the specification, then that is further
evidence that Web IDL is doing the right thing.  We might want to amend
the first criterion to say that we want at least n specifications using
Web IDL *correctly*.

We also want to know if the specification’s deep technical requirements
are appropriate for implementors.  If we want to restrict ourselves to
looking at particular specifications (like Selectors API, DOM Core, or
HTML5) then we can, assuming those other specifications are mature
enough, test whether implementations of their IDL fragments are correct.
So we might want to have an exit criterion being that we have at least n
implementations of a particular Web IDL feature that is used in
specification X.

> A test suite would need to show that the ECMAscript bindings [1]
> of Web IDL fragments actually follow the various constraints and
> algorithms that the spec defines.
>
> Ideally, the said Web IDL fragments would come (or be inspired) from
> APIs that are already well-deployed; I'm not sure if we have enough
> such APIs to cover all the features of Web IDL (but if we don't, maybe
> the features that don't match that requirement should be marked at
> risk / moved to v2?).

I think it’s a bit strange to have Web IDL’s advancement tied to whether
implementors will update their implementations of existing
specifications to follow Web IDL’s requirements if there isn’t a
specification that does recast that spec in Web IDL.

I wonder if the existing specifications that use Web IDL aren’t enough
for us to use.

> Even more ideally, in the process of building that test suite, we would
> build a tool that can generate tests to verify the correct
> implementation of the Web IDL fragment in a given browser. There is
> already at least one tool that does this [2], but it doesn't match the
> latest evolution of Web IDL (and is also probably too limited as is to
> be sufficient toward that).

If we restrict ourselves to testing particular features of Web IDL (as
used by other specifications), then I don’t think we need to have such a
tool for our test suite.  For example, if we took DOM Core as one of the
specs we were testing with, then I don’t think the Web IDL test suite
would need to test that the proprties for both Node::ELEMENT_NODE and
Node::ATTRIBUTE_NODE cannot be written to.  It’s this kind of repetition
that this tool would help you with, though.

I think the tool would be more useful to generate tests for the test
suites of those other specifications.

…
> 1. I pretty much think the Java binding should be moved out of the
> main spec into a Note; it seems to me very unlikely that anyone will
> spend the necessary effort to ensure that the Java bindings are
> correct in the timeframe we want for Web IDL progression.

That’s a good point.  I wouldn’t want it to lose its normativity,
though.  If nobody is implementing web platform specifications with
Java, then maybe it could be a separate specification that lived forever
at CR, or at least until there sufficient implementations.  (There are a
few Apache projects that implement DOM and other related things in Java;
they could count.)

-- 
Cameron McCormack ≝ http://mcc.id.au/

Received on Tuesday, 31 May 2011 04:51:25 UTC