W3C home > Mailing lists > Public > public-test-infra@w3.org > January to March 2017

Re: testharness.js v. reftests for rendering tests

From: Geoffrey Sneddon <me@gsnedders.com>
Date: Tue, 3 Jan 2017 15:08:08 +0000
Message-ID: <CAHKdfMjPmUGOSrDm_PbdzU=UiBRermv7oVYzXiDH0eQPjXRbEg@mail.gmail.com>
To: Philip J├Ągenstedt <foolip@google.com>
Cc: Simon Pieters <simonp@opera.com>, public-test-infra <public-test-infra@w3.org>
I wonder if the right middle-ground is some very generic comment about
test types: "choose between [i.e., reftests or testharness.js tests]
preferring the former for tests about layout and the latter for
everything else, but if it would be overly cumbersome to do so use the
other"?

/g

On Tue, Dec 20, 2016 at 12:51 PM, Philip J├Ągenstedt <foolip@google.com> wrote:
> Right, I have used CSSOM to test the Fullscreen UA stylesheet, that was
> useful I think. Some reftests are still good to ensure, in the case of
> Fullscreen, that the top layer stuff really is on top of everything.
>
> Maybe we should advise at least one reftest to ensure each CSS property and
> any interesting interaction, but that tests that aren't specifically trying
> to test that can use CSSOM instead?
>
> On Tue, Dec 20, 2016 at 1:19 PM Simon Pieters <simonp@opera.com> wrote:
>>
>> On Tue, 20 Dec 2016 13:03:02 +0100, Geoffrey Sneddon <me@gsnedders.com>
>> wrote:
>>
>> > Hi,
>> >
>> > As far as I'm aware, we have no real defined policy as to how rendering
>> > tests should be written. We essentially have two options: testharness.js
>> > using the CSSOM or reftests.
>> >
>> > I believe the current Blink policy is to use the former except when
>> > testing paint code, and Gecko's is to use reftests for both.
>> >
>> > On the whole, despite the performance penalty, I'd much favour
>> > recommending reftests for both given the intrinsic link between
>> > rendering and painting and the various optimisations different
>> > implementations do to avoid invoking the various parts with different
>> > mutations.
>> >
>> > Of course, if someone thinks the performance penalty is too high maybe
>> > we'll have to reconsider.
>> >
>> > /gsnedders
>> >
>>
>> I don't have a strong opinion about policy but I will point out that it
>> can sometimes be useful to test both. Equivalent CSSOM does not
>> necessarily mean equivalent rendering and vice versa.
>>
>> As an example, for testing the UA stylesheet, it seems most useful to
>> first test the CSSOM for everything. But reftests can be useful for e.g.
>> testing interaction of writing modes and form controls, or margin
>> collapsing quirks, <ol> numbering, framesets, etc.
>>
>> --
>> Simon Pieters
>> Opera Software
>>
>
Received on Tuesday, 3 January 2017 15:08:41 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 17:34:13 UTC