Re: testing plans for Web IDL

Hi Cameron,

Le mardi 31 mai 2011 à 16:50 +1200, Cameron McCormack a écrit :
> We want to know if the specification’s requirements are appropriate for
> authors of other web platform specifications.  We can see that there are
> a number of specifications that do use Web IDL, so that is some evidence
> that Web IDL is doing the right thing.  That there are more than n
> specifications using Web IDL might be one criterion for progressing past
> CR.  Should we also wait to see if there are more than n specifications
> using each feature from Web IDL?  Some features are peculiar things that
> we don’t necessarily want more specifications using, and which we have
> only for the benefit of compatibility in HTML5.  So I am not sure if
> “≥ n specs using feature X” is a good exit criterion.

I think I would go for:
* by default, each feature of Web IDL needs to be present in at least 2
specs
* except for features of Web IDL that are explicitly marked as legacy
artifacts in the spec, for which a single legacy reference would suffice

This also ensures the Web IDL spec would clearly mark which features are
to be avoided :)

I know PLH has a tool that scrapes Web IDL fragments from a number of
specs; PLH, are the said Web IDL available somewhere? This could be a
good starting point for identifying which features might not make the x2
criteria.

> Whether these other specifications write correct IDL fragments is
> another thing to look at, and something we can (and you do) test for.

Yes, I think that as part of finding specs that use the various Web IDL
features, we should also ensure that they use them properly — and by
that I mean also beyond the syntax, ensuring that the semantics of the
fragments match the prose / perceived intent of the author.

> We also want to know if the specification’s deep technical requirements
> are appropriate for implementors.  If we want to restrict ourselves to
> looking at particular specifications (like Selectors API, DOM Core, or
> HTML5) then we can, assuming those other specifications are mature
> enough, test whether implementations of their IDL fragments are correct.
> So we might want to have an exit criterion being that we have at least n
> implementations of a particular Web IDL feature that is used in
> specification X.

Yes, I think having 2 implementations of each Web IDL feature (legacy or
not) would be great.

> I think it’s a bit strange to have Web IDL’s advancement tied to whether
> implementors will update their implementations of existing
> specifications to follow Web IDL’s requirements if there isn’t a
> specification that does recast that spec in Web IDL.

Fair enough.

> I wonder if the existing specifications that use Web IDL aren’t enough
> for us to use.

If they cover enough of the required features, yes, I think that's
probably enough; we'll probably also want to have specs that are stable
enough (although of course we don't want to be trapped in an egg and
chicken problem where specs can't get stable because Web IDL isn't
finished, and Web IDL can't be finished because it needs stable specs.)

> If we restrict ourselves to testing particular features of Web IDL (as
> used by other specifications), then I don’t think we need to have such a
> tool for our test suite.  For example, if we took DOM Core as one of the
> specs we were testing with, then I don’t think the Web IDL test suite
> would need to test that the proprties for both Node::ELEMENT_NODE and
> Node::ATTRIBUTE_NODE cannot be written to.  It’s this kind of repetition
> that this tool would help you with, though.
> 
> I think the tool would be more useful to generate tests for the test
> suites of those other specifications.

Probably, indeed.

> …
> > 1. I pretty much think the Java binding should be moved out of the
> > main spec into a Note; it seems to me very unlikely that anyone will
> > spend the necessary effort to ensure that the Java bindings are
> > correct in the timeframe we want for Web IDL progression.
> 
> That’s a good point.  I wouldn’t want it to lose its normativity,
> though.  If nobody is implementing web platform specifications with
> Java, then maybe it could be a separate specification that lived forever
> at CR, or at least until there sufficient implementations.  (There are a
> few Apache projects that implement DOM and other related things in Java;
> they could count.)

I guess I wouldn't mind keeping the Java binding on the Rec track for
the time being, and moving it to Rec or to Note depending on the
interest in the document. Should the split happen before Last Call,
though?

Summarizing a possible action plan based on the discussion in this
thread:
* we need to identify all the features of Web IDL; I guess a good
starting point is the various syntactic keywords that it defines
* among them, we need to mark legacy ones
* we should start looking for specs that use the various identified
features of Web IDL, with a preference for the most mature/deployed
* we should write test cases that allows to verify the proper ECMAScript
binding of these features in the said specs

Dom

Received on Tuesday, 31 May 2011 08:28:51 UTC