Re: Ref-test image flakiness when web fonts are used

On Fri, Mar 3, 2017 at 3:48 PM, Koji Ishii <> wrote:
> Hi all, I'm new to this ML, apologies in advance if I'm not doing good,
> appreciate to point them out.
> I've been long troubled and wondering how to solve the subject, assuming
> this is our specific problem. I happened to talk to foolip about this, who
> suggested to talk to Geoffrey and James, who suggested to move the
> discussion here. So...allow me to start by forward the first a couple of
> e-mails with their approvals.

Given mine and James' emails never got forwarded, have them below
(this is my final reply, and contains the rest quoted):

---------- Forwarded message ----------
From: Geoffrey Sneddon <>
Date: Fri, Mar 3, 2017 at 1:53 PM
Subject: Re: Ref-test image flakiness when web fonts are used
To: James Graham <>
Cc: Koji Ishii <>, Dominik Röttsches
<>, Philip Jägenstedt <>,
Kunihiko Sakamoto <>

On Fri, Mar 3, 2017 at 1:39 PM, James Graham <> wrote:
> On 03/03/17 13:32, Geoffrey Sneddon wrote:
>> FWIW: this should probably be on public-test-infra, feel free to
>> forward/etc. (LMK if you're not okay with it being forwarded)
>> On Fri, Mar 3, 2017 at 7:33 AM, Koji Ishii <> wrote:
>>> Hi James, Geoffrey,
>>> We were talking about this for a while, and when I talked to foolip, he
>>> suggested to reach to you two experts. Could you mind to help us?
>>> The problem is, when we run ref-tests that use web fonts, our bots often
>>> report flakiness of images. Here's an example screenshots of a test.
>>> Probably, on heavy load or under some conditions, web fonts are not
>>> loaded
>>> yet, but test runner thinks layout is finished and takes screenshots.
>>> We haven't reached a conclusion how it happens and how to fix it, but
>>> adding
>>> a script that waits for document.fonts.ready promise to resolve after
>>> window.onload seems to work.
>>> Is this something you have seen before? Could you mind to share your
>>> thoughts?
>> Yes, this is: it's
>> <>, filed
>> by you several years ago. :)
>> There's also
>> <>
>> linked from that, though John Daggett's email it is in reply to is
>> possibly untrue now.
>> When the tests were written, there was a common belief that the
>> document's load event, as defined, should be blocked by web fonts
>> loading.
>> Now the situation is more complex; the load event is blocked by
>> "critical subresources" (in HTML) and which "resources are considered
>> critical or not is defined by the specification that defines the
>> resource's format". Except, uh, CSS doesn't define that. The HTML spec
>> used to have a bit more on this
>> (<>),
>> and per the bug linked
>> (<>) everyone
>> except WebKit had interop that background images and fonts block the
>> load event. The bug has been assigned to CSS since 2012 and seen
>> absolutely nothing happen to it since.
>> So, uh, the behaviour is totally undefined. YAY!
>> I think what we need to define for web-platform-tests is that for
>> reftests the screenshot should be taken after the last of the
>> following:
>>  * after the document's load event has been fired
>>  * the document.fonts.ready promise being fulfilled
>>  * all @imports from stylesheets have been loaded and style recomputed
>> etc.
>>  * after all <uri> subresources in all stylesheets have been loaded
>> etc. (so background-image, content, etc.)
>> I think that's it?
> The user agent has actually painted (there seems to be a bug in blink where
> we sometimes take a screenshot before images have painted, even though load
> has fired).
> Are there actually web apis to determine all these things?

No. Which yes, makes this awkward.


Received on Wednesday, 8 March 2017 16:18:22 UTC