Re: Mismatch between CSS and web-platform-tests semantics for reftests

On Aug 19, 2014, at 3:28 PM, James Graham <james@hoppipolla.co.uk> wrote:

> On 19/08/14 19:53, Peter Linss wrote:
>> I'm not sure if I'm mis-parsing your explanation or you're
>> misunderstanding the way our references work.
>> 
>> To be clear, a test can have multiple reference <links>, if so the
>> test must match one of the references.
>> 
>> A reference can <link> to additional references, in this case, the
>> test must match _all_ references in the chain.
>> 
>> Yes, the chain can form a tree (but we prefer loops).
> 
> Right, this matches my understanding of your model, although I'm not
> sure what you mean by "we prefer loops".

Say if a test has three references, a, b, & c. My preference is that the test links to a, a links to b, b links to c, and then c links back to a, forming a loop within the references.

This ensures that if another test comes along and links to b or c, they get all three references anyway. Note this is not a hard requirement and we don't test for it (but we do detect and break cycles in our testing tools).

> 
>> 
>> Both cases are necessary and used. As fantasai pointed out, in some
>> cases there are multiple possible renderings of a feature. In others,
>> it's often possible for a reference to fail in the same way that a
>> test can fail, leading to a false positive. We solve that by having
>> multiple references that use different techniques to render.
>> 
>> It's also possible to combine the two cases.
>> 
>> Then there are the mismatch references, which can be linked from the
>> test or any of the references. These are also important to detect
>> references that aren't rendering properly.
> 
> Yes, these aren't a problem.

Cool, note there are also cases with multiple mismatch references.

> 
>> I don't have usage stats of the various cases handy, but I know of a
>> bunch of tests offhand that require each scenario, and I've also
>> recently advised test authors about these features to solve problems
>> they were encountering. This isn't something we can drop or
>> significantly water down.
> 
> Can you actually point to some concrete examples? No one has done that
> so far, much less estimated how often these features are required.

Here's one:
http://test.csswg.org/shepherd/testcase/background-color-049

in this test, the two different references account for possible rounding behavior of percentage color values.

If I got my SQL-fu it looks like these are the tests in our repo now that use multiple (alternate) match references (we have a number more that use multiple mismatch references):
background-color-049
background-color-054
background-color-070
background-color-075
background-color-090
background-color-095
background-color-110
background-color-115
background-image-001
background-image-002
background-image-003
background-image-005
position-relative-001
regions-transforms-013
text-transform-capitalize-001
text-transform-lowercase-001
text-transform-uppercase-001

I haven't reviewed them all to see if they're doing it right or not, but in addition to these, fantasai already mentioned some additional features that we need to test this way that we may not have tests for yet.

> 
> Are these features something that any actual implementation is running?
> As far as I can tell from the documentation, Mozilla reftests don't
> support this feature, and I guess from Dirke's response that
> Blink/WebKit reftests don't either. That doesn't cover all possible
> implementations of course.

I'm actually in the middle of a big cleanup of our test harness and it will support this feature when I'm done (so far we haven't been able to represent the situation in our manifest files properly, I'm fixing that too).

Peter