Re: Overview of testing in view of CR exit

On 14/05/2013 18:17 , James Graham wrote:
> On 05/14/2013 04:38 PM, Robin Berjon wrote:
>> Hi,
>>
>> based on the discussion we had at the face to face,
>
> Could you summarise those discussions for those who weren't there?

Sure. The exit criteria we have are those:

 
http://dev.w3.org/html5/decision-policy/public-permissive-exit-criteria.html

These allow the group to make a judgement call about which parts of the 
spec need decent tests that can demonstrate sufficient interoperability, 
and which can be considered okay without much of a testing effort.

The general idea is that 5.0 is just one milestone in what is likely to 
be a longish list of further milestones. As such it needs to be an 
improvement on the past (I'm not too worried on that front) and a decent 
stepping stone for the next milestones (i.e. don't roger the future by 
shipping crazy unsupported stuff). In other words, we're not aiming for 
perfection, but we want to avoid making it harder to reach in future.

Based on this plan, the group picked which parts it honestly felt needed 
addressing as opposed to the bits it felt were good enough. "Good 
enough" is of course a rough metric; in my participation in the 
discussion I based my assessment on my experience in using given 
features in development (I suspect others had similar rules of thumb). 
But of course it could very well be that one's development preferences 
hides some issues away, so feedback is of course appreciated.

>>  I've made a pass
>> over the ToC to reflect the notions we had about what is considered
>> stable on its own (as per exit criteria), what requires testing, and in
>> the latter set what has implementations and/or tests (I took a
>> conservative approach to flagging that and will be refining it to add
>> more).
>
> You will need to state your assumptions more clearly if you want useful
> feedback. For example I can identify several parts that are marked as
> "interoperable" that have known interoperability problems.

Of course. The question is whether they are severe enough to warrant 
being on the priority list given the above characterisation.

> Possibly this isn't a problem; if this is just an exercise in
> getting-to-CR in which the plan is to sweep the difficult problems
> under the carpet and do the minimum amount needed to make things look
> good for Process then obviously bringing up edge cases that browsers
> might not fix quickly isn't helpful.

It's not an exercise in getting-to-CR but an exercise in 
getting-out-of-CR :)

I wouldn't characterise it as sweeping the difficult problems under the 
carpet, but I would certainly agree that the goal is emphatically not to 
address all problems in the 5.0 time frame. Those problems aren't 
buried, too; they're moved to 5.1 (which is actively worked on).

To put things differently, I believe the goal is two-fold:

   • Reaching Recommendation so that the RF policy can kick in;
   • Having that milestone, HTML 5.0, be an artefact of sufficient 
quality for real world industrial usage.

> Therefore it would be good to know
> what level of interop fail you are prepared to accept e.g. is it OK if
> things will normally work but have timing differences between browser?

Depending on the impact of timing differences, I probably wouldn't worry 
about that within this context.

> Is there some minimum level of usage in the wild that you care about
> (and how do you measure this) etc.

What do you have in mind?

-- 
Robin Berjon - http://berjon.com/ - @robinberjon

Received on Wednesday, 15 May 2013 10:30:32 UTC