Re: Followup to "Supergroups" message to AC Forum

On 6/21/2016 3:44 AM, Florian Rivoal wrote:
>> On Jun 21, 2016, at 16:20, Carine Bournez <carine@w3.org> wrote:
>>
>>
>> Hi Florian, all,
>>
>> On Tue, Jun 21, 2016 at 10:51:41AM +0900, Florian Rivoal wrote:
>>> The problem is that a WG isn't a self-contained team with a boss that can
>>> make people stick to the plan. Member companies prioritize things
>>> however they want regardless of what the charter says, and depending
>>> on the company, individuals also have quite a bit of latitude in how they
>>> prioritize things. Yes, discussing priorities may bring some ammount of
>>> alignment within the group, but if someone's priorities shift (or were never
>>> aligned in the first place), there's not much the chairs can do to force
>>> people to work according to the group's documented priorities.
>> That is more or less why spec ETAs defined in charters are never accurate.
>> (bad estimation of spec time in the first place, going back to refine spec,
>> getting implementations right, and testing). WGs are consulted in the
>> chartering process so as to get a sense of their priorities, but the charter
>> duration is longer than priorities stability. If we make shorter charters,
>> the burden of rewriting charter increases, if we make them longer, we get
>> even more inaccurate timelines. So I would advocate for external timeline
>> data that can be updated during the charter lifetime.
> Separating the Charter from the timeline sounds reasonable to me. We'll still need to figure out what the incentives are for keeping the timeline up to date and accurate though.
>
> Also, I am not sure ETAs and priorities are the same thing. It is not a rare situation to have multiple members or even the majority of the WG agree that something is high priority, and many people spend plenty of energy advancing the spec, yet nobody commits to making (or to sharing) tests. In the absence of tests, the ETA is still infinity. Is the priority low though? You could say so, since arguably, people should actually commit resources to testing when something is high priority, but by that metric nothing is high priority, making it a pretty useless concept.

This actually is a benefit of having timelines when properly used by the 
WG and Chairs.

Well in advance of charter renewal time the Chair convenes a discussion 
with the WG.  "For us to continue as a WG we should be able to have some 
sense of what we want to get done and when."  That generates a 
discussion, a consensus about priorities, and a plan (including testing) 
how to get it done.

Of course things change and W3C has never been so rigid to blow up a WG 
just because priorities changed.  But when they change, the existence of 
chartered deliverables (and a conversation that led to them) allows an 
explicit discussion.  "Oh, we are missing the plan we all agreed to.  
Why? - Our prioritized changed; let's document our new priorities.  Our 
priorities didn't change but the work slipped; let's redouble our efforts."

>
>
>> [...]
>>> The problem with testing is not merely that it is difficult to estimate. It
>>> is (at least in the WGs/CGs I am familiar with) woefully understaffed
>>> when compared to the spec-writing side. WG chairs have some ability to
>>> make their WG more or less tester friendly, but they cannot effectively
>>> designate volunteers and force the work to happen. Unless member companies
>>> commit to have people spend time on test, charter ETA estimation often
>>> boils down to "Should we pretend somebody is writing/reviewing tests for
>>> this spec, even though currently no-one is?"
>> That's a member priorities issue, not a W3C process issue.
> Sure, but we cannot have a realistic ETA for specs unless we figure out the resource assignment. so the member priorities issue is a dependency for the W3C process issue, and that's what makes it broken.

Testing plans should be discussed when creating the Charter.  That is 
one of the benefits of the Chartering process, members have an 
opportunity to plan collectively what tests they are willing to do and 
what can get done.

>
> And when I say that we need to figure out the resource assignment, I am not speaking fine grained projections. Even answers to very basic questions like "will any member company of this working group submit tests for this specification within the next 3 years", or "will anyone review these tests that have been submitted a year ago" tend to remain unanswered (not always, but often enough).
>
>> That's a bit
>> puzzling that (some, too many?) members don't value testing as a group
>> effort. Test suites can be so valuable for them when done correctly...
> Totally. I think the testing effort is probably suffering from death (or at least serious damage) by a thousand paper cuts, and we will need to address many issues before we start seeing the results we want. The CSSWG, taking lessons from WebPlatformTests, is currently changing its test submission and reviewal process to lower the friction, and that should help.
>
> Among many other things, there's also probably an issue of social prestige difference. If you do spec works, you get to attend all these nice(?) meetings and fly around the world. If you do test work, the action's over here on github.

The highest level of prestige for a developer - prominence on github ;)

>   If you lead spec work, you get to put your name at the top of a fancy document the whole world will look at. For tests, authorship can be found in commit logs... I am certainly not claiming that prestige is the main reason people join a working group or do spec work, but I would not be surprised if it did play a role in tilting the balance.

We can fix this easily.  We can (should?) put the names of the test 
leads at the top of the documents.

>
>   - Florian

Received on Tuesday, 21 June 2016 13:59:44 UTC