Re: Mismatch between CSS and web-platform-tests semantics for reftests

On Tue, Aug 19, 2014 at 5:38 PM, fantasai <fantasai.lists@inkedblade.net>
wrote:

> On 08/19/2014 03:28 PM, James Graham wrote:
>
>>
>>  I don't have usage stats of the various cases handy, but I know of a
>>> bunch of tests offhand that require each scenario, and I've also
>>> recently advised test authors about these features to solve problems
>>> they were encountering. This isn't something we can drop or
>>> significantly water down.
>>>
>>
>> Can you actually point to some concrete examples? No one has done that
>> so far, much less estimated how often these features are required.
>>
>> Are these features something that any actual implementation is running?
>> As far as I can tell from the documentation, Mozilla reftests don't
>> support this feature, and I guess from Dirke's response that
>> Blink/WebKit reftests don't either. That doesn't cover all possible
>> implementations of course.
>>
>
> Mozilla uses the multiple-references-that-must-all-match feature.
> No implementation supports match-any-of-these-references because
> for a given implementation, only one answer is chosen and therefore
> the others become mismatch references for that implementation. But
> for the shared test suite, we need to have that semantic.


Okay, I think Peter's response to James just now actually answers my
question. If test X links to reference A, ref A links to ref B,  then you
really are actually testing the references themselves, right? (which is
fine, I just wondered if I was missing something, earlier ...).

Put differently, in order to test if X rendered properly, you only need to
compare it to A if you can assume A is correct. You can, separately, test A
for correctness by comparing to B (and so forth), of course.

'match-any-of-these' is a different problem; unless you know ahead of time
*which* reference is supposed to match for a given implementation, you
would actually need to check all of the references, right?

(In theory Blink's test harness supports match-any, but I don't think we
use it in practice).

-- Dirk