Re: Ref-test image flakiness when web fonts are used

Ping Geoffrey. Koji and I were just chatting and it seems reasonable to
have reftests wait for webfonts by default, as long as there is an escape
hatch so that it's possible to test with slow-loading fonts or what happens
if a first font load fails and then you try to load another font. Would it
be weird to treat class="reftest-wait" as that escape hatch, or do we need
something else?

Practically speaking, how do we go about proclaiming that reftests now
should wait for document.fonts.ready? Is it a matter of changing wptrunner
and filing browser bugs that they probably want to do something similar?

On Mon, Mar 20, 2017 at 3:54 PM Philip Jägenstedt <foolip@chromium.org>
wrote:

> I guess that would be the WebDriver spec, is that right Geoffrey, or is
> there a "spec" for what things reftests should wait for specifically?
>
> What is rendered while web fonts are loading is something web authors have
> to care about, would it still be possible to test this somehow? Currently,
> you could do it with a slow-loading font resource using wptserve.
>
> On Wed, Mar 15, 2017 at 12:36 PM Koji Ishii <kojii@chromium.org> wrote:
>
> Thank you Geoffrey, that'd be great.
>
> Could you mind to tell me which spec should this go? I'm not familiar with
> specs for test runner.
>
> Should I file a github issue for the spec?
>
> 2017/03/15 午前2:02 "Geoffrey Sneddon" <me@gsnedders.com>:
>
> On Tue, Mar 14, 2017 at 7:38 AM, Koji Ishii <kojii@chromium.org> wrote:
> > Apologies that I was late to reply to the thread I asked...
> >
> > Thank you Geoffrey for sharing the background, and I'm surprised I'm
> > sometimes there ;-O
> >
> > On Thu, Mar 9, 2017 at 1:17 AM, Geoffrey Sneddon <me@gsnedders.com>
> wrote:
> >
> >> >> I think what we need to define for web-platform-tests is that for
> >>
> >> >> reftests the screenshot should be taken after the last of the
> >> >> following:
> >> >>
> >> >>  * after the document's load event has been fired
> >> >>  * the document.fonts.ready promise being fulfilled
> >> >>  * all @imports from stylesheets have been loaded and style
> recomputed
> >> >> etc.
> >> >>  * after all <uri> subresources in all stylesheets have been loaded
> >> >> etc. (so background-image, content, etc.)
> >> >>
> >> >> I think that's it?
> >
> >
> > Yes, I think the list is correct. The font loading spec says the
> > document.fonts.ready promise implies layout completes, if my reading is
> > correct.
> >
> >> > The user agent has actually painted (there seems to be a bug in blink
> >> > where
> >> > we sometimes take a screenshot before images have painted, even though
> >> > load
> >> > has fired).
> >
> >
> > Yeah, obviously we need to wait for paint, but I think it's even less
> > defined and obvious?
> >
> >>
> >> > Are there actually web apis to determine all these things?
> >>
> >> No. Which yes, makes this awkward.
> >
> >
> > So...is this something we can define or recommend somewhere? It won't be
> a
> > great situation where one vendor assumes that tests include
> > "/common/reftest-wait-for-fonts.js", while another vendor assumes the
> test
> > runner waits for these condition, no?
>
> I think we should simply define that the test runner waits for these
> conditions: I don't think we want to make every single reftest that
> relies on web fonts to have to have a bunch of JS in it (because it'll
> be virtually *all* such reftests), as there will inevitably be some
> where it gets forgotten. The perf hit in the common case (with no web
> fonts) seems like it should be sufficiently small to not be an issue.
>
> /g
>
>