Re: Testing Challenges -- Re: Followup to "Supergroups" message to AC Forum

Sorry, sent too early.  I intended to add:

Jeff rightly points out that W3C has tried all sorts of approaches to getting the testing done that’s needed to get a spec to Rec.  I’d argue that its limited success is  because “testing” is a tangle of issues, including:


1.       Testing the spec itself – is it clear enough so that different implementations based on the spec itself CAN interoperate ?

2.       Testing an implementation – does it conform to the spec?

3.       Testing the ecosystem – Is there real world interoperability on the features described by the spec (irrespective of whether the spec is clear or the implementations conformant)?

4.       What features really interoperate?  This is what caniuse.com tries to describe, not sure how rigorous the testing behind it is.

Any strategy to address the testing problem needs to be explicit about which of these the strategy is trying to address.  I’d argue that #1 is clearly in W3C’s domain and would advocate using more staff resources for this kind of testing.  #2 is not W3C’s concern, it’s the concern of implementers and their customers. #3 gets into the equally thorny Maintenance Challenge, keeping specs up to date with what implementations actually do. #4 is interesting, and at least one CG is working in this area, but I’m not sure it’s relevant to the W3C process.


From: Michael Champion <Michael.Champion@microsoft.com>
Date: Tuesday, June 21, 2016 at 9:03 AM
To: "jeff@w3.org" <jeff@w3.org>, Daniel Glazman <daniel.glazman@disruptive-innovations.com>, "public-w3process@w3.org" <public-w3process@w3.org>
Subject: Testing Challenges -- Re: Followup to "Supergroups" message to AC Forum
Resent-From: <public-w3process@w3.org>
Resent-Date: Tuesday, June 21, 2016 at 9:04 AM


From: "jeff@w3.org" <jeff@w3.org>
Date: Tuesday, June 21, 2016 at 8:56 AM
To: Daniel Glazman <daniel.glazman@disruptive-innovations.com>, "public-w3process@w3.org" <public-w3process@w3.org>
Subject: Re: Followup to "Supergroups" message to AC Forum
Resent-From: <public-w3process@w3.org>
Resent-Date: Tuesday, June 21, 2016 at 8:56 AM


I'm happy to entertain new strategies to demonstrate two interoperable implementations.  Here are some choices (most of whom we've tried already at some level).

  *   Encourage open source contributions (e.g. TTWF)
  *   Test frameworks
  *   Vendors providing their tests
  *   Funding set aside for testing
  *   Rely to some extent on interoperability being demonstrated in the marketplace