W3C home > Mailing lists > Public > public-test-infra@w3.org > July to September 2014

Re: Mismatch between CSS and web-platform-tests semantics for reftests

From: Dirk Pranke <dpranke@chromium.org>
Date: Tue, 19 Aug 2014 17:55:13 -0700
Message-ID: <CAEoffTDFktxDyW8qKvz+JmzO6+7KH0-hs62HxxC2gP-8pPgLwg@mail.gmail.com>
To: fantasai <fantasai.lists@inkedblade.net>
Cc: James Graham <james@hoppipolla.co.uk>, public-test-infra <public-test-infra@w3.org>
On Tue, Aug 19, 2014 at 5:38 PM, fantasai <fantasai.lists@inkedblade.net>

> On 08/19/2014 03:28 PM, James Graham wrote:
>>  I don't have usage stats of the various cases handy, but I know of a
>>> bunch of tests offhand that require each scenario, and I've also
>>> recently advised test authors about these features to solve problems
>>> they were encountering. This isn't something we can drop or
>>> significantly water down.
>> Can you actually point to some concrete examples? No one has done that
>> so far, much less estimated how often these features are required.
>> Are these features something that any actual implementation is running?
>> As far as I can tell from the documentation, Mozilla reftests don't
>> support this feature, and I guess from Dirke's response that
>> Blink/WebKit reftests don't either. That doesn't cover all possible
>> implementations of course.
> Mozilla uses the multiple-references-that-must-all-match feature.
> No implementation supports match-any-of-these-references because
> for a given implementation, only one answer is chosen and therefore
> the others become mismatch references for that implementation. But
> for the shared test suite, we need to have that semantic.

Okay, I think Peter's response to James just now actually answers my
question. If test X links to reference A, ref A links to ref B,  then you
really are actually testing the references themselves, right? (which is
fine, I just wondered if I was missing something, earlier ...).

Put differently, in order to test if X rendered properly, you only need to
compare it to A if you can assume A is correct. You can, separately, test A
for correctness by comparing to B (and so forth), of course.

'match-any-of-these' is a different problem; unless you know ahead of time
*which* reference is supposed to match for a given implementation, you
would actually need to check all of the references, right?

(In theory Blink's test harness supports match-any, but I don't think we
use it in practice).

-- Dirk
Received on Wednesday, 20 August 2014 00:56:06 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:11 UTC