Re: Followup to "Supergroups" message to AC Forum

On 21/06/2016 09:44, Florian Rivoal wrote:

> Separating the Charter from the timeline sounds reasonable to me. We'll still need to figure out what the incentives are for keeping the timeline up to date and accurate though.
> 
> Also, I am not sure ETAs and priorities are the same thing. It is not a rare situation to have multiple members or even the majority of the WG agree that something is high priority, and many people spend plenty of energy advancing the spec, yet nobody commits to making (or to sharing) tests. In the absence of tests, the ETA is still infinity. Is the priority low though? You could say so, since arguably, people should actually commit resources to testing when something is high priority, but by that metric nothing is high priority, making it a pretty useless concept.

Exactly. An unfortunate situation but the reality.

> Sure, but we cannot have a realistic ETA for specs unless we figure out the resource assignment. so the member priorities issue is a dependency for the W3C process issue, and that's what makes it broken.

Right.

> And when I say that we need to figure out the resource assignment, I am not speaking fine grained projections. Even answers to very basic questions like "will any member company of this working group submit tests for this specification within the next 3 years", or "will anyone review these tests that have been submitted a year ago" tend to remain unanswered (not always, but often enough).

Members are not able (or not willing) to share their priorities at
3 months, how could they do it for 3 years...

>> That's a bit 
>> puzzling that (some, too many?) members don't value testing as a group
>> effort. Test suites can be so valuable for them when done correctly...
> 
> Totally. I think the testing effort is probably suffering from death (or at least serious damage) by a thousand paper cuts, and we will need to address many issues before we start seeing the results we want. The CSSWG, taking lessons from WebPlatformTests, is currently changing its test submission and reviewal process to lower the friction, and that should help.

We've been discussing the Testing issue forever and it's time to
acknowledge the fact Working Groups and WG Chairs have no control
on the CR->PR phase since it mostly depends on the availability of
a Test Suite and on implementation experience, contributions fully
in the hands of the Membership.

> Among many other things, there's also probably an issue of social prestige difference. If you do spec works, you get to attend all these nice(?) meetings and fly around the world. If you do test work, the action's over here on github. If you lead spec work, you get to put your name at the top of a fancy document the whole world will look at. For tests, authorship can be found in commit logs... I am certainly not claiming that prestige is the main reason people join a working group or do spec work, but I would not be surprised if it did play a role in tilting the balance.

I'm not sure this is the real problem behind Testing. First, browser
vendors have internal tests, that are often not easily portable to
our own testing architecture; second, there is the "good enough" issue
and the pressure from Marketing and the public to release features that
are still experimental (since they're still in CR). But releasing a
feature behind a prefix or (these days) a flag is not enough for real
mass feedback. Hence requests from the Membership to unprefix/unflag
new features. And once it's unprefixed/unflagged, it's a 'de facto'
standard anyway.

One thing remains and Florian is right there: writing tests is less
fun that writing a spec or coding and Product Managers tend to allocate
time for internal tests but rarely for W3C tests.

I think we made two mistakes here:

1. when we collapsed LCWD and CR into one single step, we should have
   kept WD and not R. The words "Candidate Recommendation" give a
   signal the document is approaching completion, which is clearly not
   the case if nobody's working on a Test Suite.

2. we are extremely bad at regressing documents on the REC track.
   There should be a six months limit to CR on the REC track and any
   document reaching six months + one day should automatically go back
   to WD if the review to PR is not already started. All documents,
   automatically, no exception.

</Daniel>

Received on Tuesday, 21 June 2016 08:07:53 UTC