Re: Ref-test image flakiness when web fonts are used

On Fri, Mar 3, 2017 at 3:48 PM, Koji Ishii <kojii@chromium.org> wrote:
> Hi all, I'm new to this ML, apologies in advance if I'm not doing good,
> appreciate to point them out.
>
> I've been long troubled and wondering how to solve the subject, assuming
> this is our specific problem. I happened to talk to foolip about this, who
> suggested to talk to Geoffrey and James, who suggested to move the
> discussion here. So...allow me to start by forward the first a couple of
> e-mails with their approvals.

Given mine and James' emails never got forwarded, have them below
(this is my final reply, and contains the rest quoted):




---------- Forwarded message ----------
From: Geoffrey Sneddon <me@gsnedders.com>
Date: Fri, Mar 3, 2017 at 1:53 PM
Subject: Re: Ref-test image flakiness when web fonts are used
To: James Graham <james@hoppipolla.co.uk>
Cc: Koji Ishii <kojii@chromium.org>, Dominik Röttsches
<drott@chromium.org>, Philip Jägenstedt <foolip@chromium.org>,
Kunihiko Sakamoto <ksakamoto@chromium.org>


On Fri, Mar 3, 2017 at 1:39 PM, James Graham <james@hoppipolla.co.uk> wrote:
> On 03/03/17 13:32, Geoffrey Sneddon wrote:
>>
>> FWIW: this should probably be on public-test-infra, feel free to
>> forward/etc. (LMK if you're not okay with it being forwarded)
>>
>> On Fri, Mar 3, 2017 at 7:33 AM, Koji Ishii <kojii@chromium.org> wrote:
>>>
>>> Hi James, Geoffrey,
>>>
>>> We were talking about this for a while, and when I talked to foolip, he
>>> suggested to reach to you two experts. Could you mind to help us?
>>>
>>> The problem is, when we run ref-tests that use web fonts, our bots often
>>> report flakiness of images. Here's an example screenshots of a test.
>>> Probably, on heavy load or under some conditions, web fonts are not
>>> loaded
>>> yet, but test runner thinks layout is finished and takes screenshots.
>>>
>>> We haven't reached a conclusion how it happens and how to fix it, but
>>> adding
>>> a script that waits for document.fonts.ready promise to resolve after
>>> window.onload seems to work.
>>>
>>> Is this something you have seen before? Could you mind to share your
>>> thoughts?
>>
>>
>> Yes, this is: it's
>> <https://bugs.chromium.org/p/chromium/issues/detail?id=507054>, filed
>> by you several years ago. :)
>>
>> There's also
>> <https://lists.w3.org/Archives/Public/www-style/2015Jun/0372.html>
>> linked from that, though John Daggett's email it is in reply to is
>> possibly untrue now.
>>
>> When the tests were written, there was a common belief that the
>> document's load event, as defined, should be blocked by web fonts
>> loading.
>>
>> Now the situation is more complex; the load event is blocked by
>> "critical subresources" (in HTML) and which "resources are considered
>> critical or not is defined by the specification that defines the
>> resource's format". Except, uh, CSS doesn't define that. The HTML spec
>> used to have a bit more on this
>>
>> (<https://github.com/whatwg/html/commit/89cff9a51f24174c2efca73fa2964e6cd258c541>),
>> and per the bug linked
>> (<https://www.w3.org/Bugs/Public/show_bug.cgi?id=17011>) everyone
>> except WebKit had interop that background images and fonts block the
>> load event. The bug has been assigned to CSS since 2012 and seen
>> absolutely nothing happen to it since.
>>
>> So, uh, the behaviour is totally undefined. YAY!
>>
>> I think what we need to define for web-platform-tests is that for
>> reftests the screenshot should be taken after the last of the
>> following:
>>
>>  * after the document's load event has been fired
>>  * the document.fonts.ready promise being fulfilled
>>  * all @imports from stylesheets have been loaded and style recomputed
>> etc.
>>  * after all <uri> subresources in all stylesheets have been loaded
>> etc. (so background-image, content, etc.)
>>
>> I think that's it?
>
>
> The user agent has actually painted (there seems to be a bug in blink where
> we sometimes take a screenshot before images have painted, even though load
> has fired).
>
> Are there actually web apis to determine all these things?

No. Which yes, makes this awkward.

/g

Received on Wednesday, 8 March 2017 16:18:22 UTC