Re: Ref-test image flakiness when web fonts are used

On Tue, Mar 14, 2017 at 7:38 AM, Koji Ishii <> wrote:
> Apologies that I was late to reply to the thread I asked...
> Thank you Geoffrey for sharing the background, and I'm surprised I'm
> sometimes there ;-O
> On Thu, Mar 9, 2017 at 1:17 AM, Geoffrey Sneddon <> wrote:
>> >> I think what we need to define for web-platform-tests is that for
>> >> reftests the screenshot should be taken after the last of the
>> >> following:
>> >>
>> >>  * after the document's load event has been fired
>> >>  * the document.fonts.ready promise being fulfilled
>> >>  * all @imports from stylesheets have been loaded and style recomputed
>> >> etc.
>> >>  * after all <uri> subresources in all stylesheets have been loaded
>> >> etc. (so background-image, content, etc.)
>> >>
>> >> I think that's it?
> Yes, I think the list is correct. The font loading spec says the
> document.fonts.ready promise implies layout completes, if my reading is
> correct.
>> > The user agent has actually painted (there seems to be a bug in blink
>> > where
>> > we sometimes take a screenshot before images have painted, even though
>> > load
>> > has fired).
> Yeah, obviously we need to wait for paint, but I think it's even less
> defined and obvious?
>> > Are there actually web apis to determine all these things?
>> No. Which yes, makes this awkward.
> this something we can define or recommend somewhere? It won't be a
> great situation where one vendor assumes that tests include
> "/common/reftest-wait-for-fonts.js", while another vendor assumes the test
> runner waits for these condition, no?

I think we should simply define that the test runner waits for these
conditions: I don't think we want to make every single reftest that
relies on web fonts to have to have a bunch of JS in it (because it'll
be virtually *all* such reftests), as there will inevitably be some
where it gets forgotten. The perf hit in the common case (with no web
fonts) seems like it should be sufficiently small to not be an issue.


Received on Tuesday, 14 March 2017 17:03:14 UTC