Re: Followup to "Supergroups" message to AC Forum

On Tue, Jun 21, 2016 at 4:56 PM, Jeff Jaffe <jeff@w3.org> wrote:
>
>
> On 6/21/2016 10:56 AM, Daniel Glazman wrote:
>
> > On 21/06/2016 16:04, Jeff Jaffe wrote:
> >
> > > Yes testing is in the hands of the Membership.  That suggests to me that
> > > we should get ahead of the game with early discussions with key
> > > companies to get testing commitments.
> >
> > We spent the last 15 years trying that, it never worked. From time
> > to time, we're lucky and our tests needs match the strategy of one
> > given vendor. But in the general case, no luck. Years of delay for
> > a spec _only_ because nobody (even the spec editor) wants to contribute
> > tests is just another day at the office for us.
> >
> > I respectfully suggest this method is not productive enough and we
> > need another strategy.
>
>
> I'm happy to entertain new strategies to demonstrate two interoperable
> implementations.  Here are some choices (most of whom we've tried already at
> some level).
>
> Encourage open source contributions (e.g. TTWF)
> Test frameworks
> Vendors providing their tests
> Funding set aside for testing
> Rely to some extent on interoperability being demonstrated in the
> marketplace

In reality, vendors providing their tests is the same as it being "in
the hands of the Membership".

I disagree, however, with Daniel's assertion that it has never
worked—I think web-platform-tests has shown that it can entirely work
(okay, web-platform-tests isn't just the Membership, but you have to
go a long way down the list of contributors to find any not paid by a
Member). To claim that we still have a serious problem is to deny six
years of work there—we have had numerous specs get out of CR with
almost no delay using tests in web-platform-tests. If there's one
lesson to be had from that, it's making it easy to submit tests means
it's easy to get tests, at least for any new feature.

Perhaps Daniel's experience is tainted by the issues the CSS WG had in
attracting test submissions, especially for 2.1, but I think there's a
lot of reasons why the CSS WG had so much trouble. 2.1 was problematic
given most vendors don't have nicely organised test suites for the
older standards to release in the first place, and later standards
have been problematic because of the CSS WG's until recent policies on
metadata which therefore vastly increased the cost of releasing test
cases to what Members obviously found unacceptably high for the
benefit it got them (the benefit was low for all but the smallest
vendors, and the cost was relatively high).

If we want to do anything new, we should consider the fact that it's
taken six years thus far to reach a point where half the major browser
vendors are running web-platform-tests regularly (and we're likely to
reach a majority within the next year), and starting from scratch will
start that all over again—and there's likely to be more resistance as
it's essentially duplicating something we already have.

/Geoffrey

Received on Tuesday, 21 June 2016 22:00:33 UTC