Re: testing plans for Web IDL

> I know PLH has a tool that scrapes Web IDL fragments from a number of
> specs; PLH, are the said Web IDL available somewhere? This could be a
> good starting point for identifying which features might not make the x2
> criteria.

FYI, we've been using the following perl script for some time:

  http://code.google.com/p/es-operating-system/source/browse/trunk/esidl/dom/idl_scraper.pl

This script works fine with specs edited by Anne, Hixie, and a few others.

- Shiki

2011/5/31 Dominique Hazael-Massieux <dom@w3.org>:
> Hi Cameron,
>
> Le mardi 31 mai 2011 à 16:50 +1200, Cameron McCormack a écrit :
>> We want to know if the specification’s requirements are appropriate for
>> authors of other web platform specifications.  We can see that there are
>> a number of specifications that do use Web IDL, so that is some evidence
>> that Web IDL is doing the right thing.  That there are more than n
>> specifications using Web IDL might be one criterion for progressing past
>> CR.  Should we also wait to see if there are more than n specifications
>> using each feature from Web IDL?  Some features are peculiar things that
>> we don’t necessarily want more specifications using, and which we have
>> only for the benefit of compatibility in HTML5.  So I am not sure if
>> “≥ n specs using feature X” is a good exit criterion.
>
> I think I would go for:
> * by default, each feature of Web IDL needs to be present in at least 2
> specs
> * except for features of Web IDL that are explicitly marked as legacy
> artifacts in the spec, for which a single legacy reference would suffice
>
> This also ensures the Web IDL spec would clearly mark which features are
> to be avoided :)
>
> I know PLH has a tool that scrapes Web IDL fragments from a number of
> specs; PLH, are the said Web IDL available somewhere? This could be a
> good starting point for identifying which features might not make the x2
> criteria.
>
>> Whether these other specifications write correct IDL fragments is
>> another thing to look at, and something we can (and you do) test for.
>
> Yes, I think that as part of finding specs that use the various Web IDL
> features, we should also ensure that they use them properly — and by
> that I mean also beyond the syntax, ensuring that the semantics of the
> fragments match the prose / perceived intent of the author.
>
>> We also want to know if the specification’s deep technical requirements
>> are appropriate for implementors.  If we want to restrict ourselves to
>> looking at particular specifications (like Selectors API, DOM Core, or
>> HTML5) then we can, assuming those other specifications are mature
>> enough, test whether implementations of their IDL fragments are correct.
>> So we might want to have an exit criterion being that we have at least n
>> implementations of a particular Web IDL feature that is used in
>> specification X.
>
> Yes, I think having 2 implementations of each Web IDL feature (legacy or
> not) would be great.
>
>> I think it’s a bit strange to have Web IDL’s advancement tied to whether
>> implementors will update their implementations of existing
>> specifications to follow Web IDL’s requirements if there isn’t a
>> specification that does recast that spec in Web IDL.
>
> Fair enough.
>
>> I wonder if the existing specifications that use Web IDL aren’t enough
>> for us to use.
>
> If they cover enough of the required features, yes, I think that's
> probably enough; we'll probably also want to have specs that are stable
> enough (although of course we don't want to be trapped in an egg and
> chicken problem where specs can't get stable because Web IDL isn't
> finished, and Web IDL can't be finished because it needs stable specs.)
>
>> If we restrict ourselves to testing particular features of Web IDL (as
>> used by other specifications), then I don’t think we need to have such a
>> tool for our test suite.  For example, if we took DOM Core as one of the
>> specs we were testing with, then I don’t think the Web IDL test suite
>> would need to test that the proprties for both Node::ELEMENT_NODE and
>> Node::ATTRIBUTE_NODE cannot be written to.  It’s this kind of repetition
>> that this tool would help you with, though.
>>
>> I think the tool would be more useful to generate tests for the test
>> suites of those other specifications.
>
> Probably, indeed.
>
>> …
>> > 1. I pretty much think the Java binding should be moved out of the
>> > main spec into a Note; it seems to me very unlikely that anyone will
>> > spend the necessary effort to ensure that the Java bindings are
>> > correct in the timeframe we want for Web IDL progression.
>>
>> That’s a good point.  I wouldn’t want it to lose its normativity,
>> though.  If nobody is implementing web platform specifications with
>> Java, then maybe it could be a separate specification that lived forever
>> at CR, or at least until there sufficient implementations.  (There are a
>> few Apache projects that implement DOM and other related things in Java;
>> they could count.)
>
> I guess I wouldn't mind keeping the Java binding on the Rec track for
> the time being, and moving it to Rec or to Note depending on the
> interest in the document. Should the split happen before Last Call,
> though?
>
> Summarizing a possible action plan based on the discussion in this
> thread:
> * we need to identify all the features of Web IDL; I guess a good
> starting point is the various syntactic keywords that it defines
> * among them, we need to mark legacy ones
> * we should start looking for specs that use the various identified
> features of Web IDL, with a preference for the most mature/deployed
> * we should write test cases that allows to verify the proper ECMAScript
> binding of these features in the said specs
>
> Dom
>
>
>
>
>
>

Received on Tuesday, 31 May 2011 08:53:01 UTC