Re: WebPerfWG call - June 18th @ 10am PST

Hi everyone,

I want to do a correction on what I presented on the call regarding Origin
Policy. Looks like it is currently blocked on some design questions, and as
far as I could tell there is currently no one attempting to drive the
proposal to completion... Therefore, we might want to revisit the
dependency of an Abandonment API on it.

On Tue, Jun 30, 2020 at 12:27 PM Yoav Weiss <yoav@yoav.ws> wrote:

> Hey folks,
>
> Minutes
> <https://docs.google.com/document/d/e/2PACX-1vTZv8pNLfaf1r6tSvmkNqAQxmAk8k81y46LbrxcGCeBR4B9HWAy8HNhzEg3y-hf4Plg15M58ettqT80/pub>
> and video <https://www.youtube.com/watch?v=Z7Sgk9E5ikQ> from the call are
> now (belatedly) available. Copying them below for safe keeping.
>
>
> WebPerfWG call - June 18th 2020
> Updated automatically every 5 minutes
>  Participants
>
>    - Noam Helfman, Yoav Weiss, Nic Jansma, Steven Bougon, Thomas Kelly,
>    Gilles Dubuc, Annie Sullivan, Ian Clelland, Michal Mocny, Michelle Vu,
>    Nicolás Peña, Dominic Farolino, Benjamin De Kosnik, Alex Christensen, Ilya
>    Grigorik, Philip Walton
>
> Next Meeting
>
> July 2nd @ 10am PST / 1pm EST
> PresentationsAbandonment (explainer
> <https://www.google.com/url?q=https://github.com/anniesullie/web-page-abandonment-api/blob/master/README.md&sa=D&ust=1593537574147000&usg=AOvVaw1krEN8hmYt5WRBLsqCwTU9>
> , slides
> <https://www.google.com/url?q=https://docs.google.com/presentation/d/1Kvm8YsYA3yHgiRh-_ZFf6jeSc-GP1qGOVqXSVeNzVE4/edit%23slide%3Did.p&sa=D&ust=1593537574148000&usg=AOvVaw3ae6zkvnw6oDd3xoAkEVeR>
> )
>
>    - Nicolás: We've talked about abandonment for a while
>    - ... Want to enable analytics providers and site owners to get
>    visibility about users that leave the page early (before analytics have had
>    time to register themselves)
>    - … that’s a current blind spot
>    - ... Want sites to be able to correlate against business metrics
>    - ... Abandon definition:
>
>
>    - User has explicitly navigated away from the page (redirects don't
>    count)
>    - Page did not reach First Contentful Paint (FCP)
>    - Page was in the foreground at some point
>
>
>    - ... Non-abandons:
>
>
>    - User explicitly navigates away
>    - Page reached FCP
>    - Page was in foreground at some point
>
>
>    - ... Need non-abandons to have denominator, as some browsers won't
>    support this API and analytics needs to know abandon rate
>    - ... Why FCP? If not reached, the user didn't get value. Reaching FCP
>    doesn’t guarantee user value, but understanding user value is beyond scope
>    for this proposal.
>    - ... Not using onload (unrelated to user experience), LCP (we don’t
>    know we reached LCP when the user abandons) or lack of input (there are
>    pages that don’t require interaction)
>    - ... What should we expose? Counts for abandons and non-abandons
>    - ... Maybe: navstart-to-abandon-time, foreground-duration
>    - ... How to expose this information?  Reporting API.
>    Network-centered reporting (via HTTP headers)
>
>
>    - Would be better to base that on Origin Policy, but unclear when it
>    will be ready
>    - NEL-like headers only solution won’t capture first-view cases where
>    abandonment happened before headers arrived
>
>
>    - Ilya: NEL currently explicitly excludes user-abandoned actions such
>    as back-button or tab close
>    - Nic: Would this be "sticky" on the domain, where once you register,
>    it would apply to future navigations?
>    - Ian: Yes, and it's per-origin not per-file
>    - Steven: Can you go over how abandonment might be useful?
>    - Nicolás: Let's say you have a normally operating website, and FCP
>    improves.  Is it because users were abandoning in the worse-case scenarios?
>
>
>    - Also, if you see a smaller number of users reported - is it because
>    more users abandon before you register them?
>
>
>    - ... Can also correlate to business metrics
>    - Ilya: There's a non-trivial amount of abandoned navigations across
>    the web, where there's no visibility today. We want to expose that.
>    - Annie: 6-8% of navigations are abandoned on mobile, varies by site.
>    Sites that didn’t meet Core Web Vitals were 24% more likely to be abandoned.
>    - Ben: Does the 6-8% account for network errors or just abandons?
>    - Annie: Just user abandons, excluded redirects or other detectable
>    network errors in the numerator or denominator
>    - Michelle: From Pinterest, we are interested in this as we've just
>    done an abandonment analysis, but are probably missing cases that don’t
>    register.  Were there other metrics besides FCP considered, like TTI?
>    - Annie: TTI might be too late, LCP is a continuous metric that may
>    always increase over time
>    - Yoav: One thought is to have FCP as default, but the website could
>    declare a certain JS variable that if it was never defined by JS, the page
>    was abandoned. For the site to say what it considers a non-abandonment
>    - Nicolás: We could definitely have a parameter that enables sites
>    measure their own definition of abandonment
>    - Philip: Could we report abandons that hadn't reached onload versus
>    abandons that haven't reached FCP? Because mostly by onload the page is
>    configured and tracked by other analytics
>    - Nicolás: What are we trying to incentivize. Onload can be made
>    arbitrarily fast, but that doesn’t improve the UX.
>    - … Worth considering if there’s another point in time where we’re
>    sure analytics are registered, but don’t know what that point in time would
>    be. Might be better to have it be site provided.
>    - Philip: Scenario where a user sees a loading screen, but it hasn't
>    reached onload or loaded analytics
>    - Nicolás: Could that be covered by the site-provided proposal?
>    - Ilya: Might be worthwhile to split up 2 cases - Up to FCP analytics
>    could not have run.  After FCP, the page is being constructed but analytics
>    may be running.
>    - Michal: What Michelle was pointing out was more along User
>    Experience abandons, that they want to measure when visitor is interacting
>    with the page and abandons it.
>    - ... Which is separate from the un-measurable experiences today
>    (before FCP was reached). Maybe worthwhile to use a different name than
>    “abandonment”
>    - Annie: There’s also “cart abandonment”
>    - Ilya: One question was dependency on Reporting API / origin-policy
>    and how that would affect our timeline
>    - Nicolás: Origin-policy should be going forward and not affect our
>    timeline
>
> Back-Forward Cache and RUM metrics
>
>    - Back-forward cache and RUM metrics
>    <https://www.google.com/url?q=https://docs.google.com/presentation/d/11MTKvv9tTXqz-atdLQCq1d_Hnoyo0VOtvZf4NpkudTA/edit%23slide%3Did.p&sa=D&ust=1593537574153000&usg=AOvVaw34Bxspsel1oYhv9xUphMlt>
>    - Yoav: We've talked about this in the past but didn't really reach
>    any conclusions
>    - ... Browser may implement or improve bfcache
>    - ... Sites can change to be "BF Cache eligible" by changing event
>    registrations (unload)
>    - ... Can result in user experience changes
>    - ... What can we do about it?
>    - ... Update old metrics?   Seems destructive.  Might not be
>    web-compatible for current analytics scripts.
>    - ... Fire new metrics?  For those we care about, and have analytics
>    scripts capture those new entries.
>    - ... Which metrics?  NavigationTiming, Paint Timings.
>
>
>    - But we’d also need to correlate other metrics to page loads.
>
>
>    - ... Web compat: Size of NavigationTiming timing array today is
>    usually 1 and if we add new entries it may cause compat concerns.
>    - ... Similar for Paint Timings.
>    - ... Correlate ResourceTiming, ElementTiming, EventTiming, Layout
>    Instability to page loads.  Need to be able to split those out from one
>    another.
>    - ... Multiple Time Origins?  All metrics are based out of a single
>    time origin today.  Should we have a metric that shows after N milliseconds
>    in, this is a new page load.  Allow for filtering of entries by navigation.
>    - Nic: Would each new NavigationTiming entry be the "start" of each
>    new time origin
>    - Yoav: That would be possible, or something similar with Visibility
>    Observer
>    - ... Right now the method to know whether your page has gone into or
>    out of bfcache is through the pageshow / pagehide events, which aren't
>    necessarily integrated into the Observer interfaces.
>    - ... The other thing we need to consider is SPA navigations.  Do they
>    affect time origins?
>    - Ilya: BFCache is new to Chromium, but not other engines. How do
>    existing analytics providers treat it?
>    - Nic: mPulse doesn’t measure it today
>    - Michal: Do existing analytics providers take visibility state into
>    account today?
>    - Nic: For mPulse/Boomerang, we track visibility state (and last time
>    it was hidden/visible) on the beacon, and let our customer segment their
>    data on that state
>    - Steven: Yes visibility, no BFCache
>    - Michal: Like the comparison to visibility. What about background
>    tabs?
>
>
>    - If you have separate “foreground sessions” to a certain tab, does
>    the analytics get split up?
>
>
>    - Nic: mPulse tracks visibility state and customers can split their
>    data based on that. Most people don’t, but we’re currently promoting that.
>    - Yoav: Folks running web properties - is this something you consider
>    today?
>    - Steven: Not BF cache, but we do consider visibility.
>    - Michelle: Same here, we just started taking visibility into account
>    last year
>    - Gilles: Same. But we don’t like the current approach to visibility
>    of simply filtering it out. Would be better to still measure those
>    scenarios and be able to identify them.
>    - Philip: Our concern is that when Chrome ships BFCache, things that
>    used to be back-forward navigations that were fast, will no longer be full
>    navigations, so those won't be represented in metrics.
>    - …  i.e. compared to other browsers, today Chrome reports additional
>    navigations because there's no BFCache, but that will soon change.
>    - ... How do existing analytics providers show data for BFCache
>    navigations?
>    - Nic: I can only remember a few times our customers have asked about
>    BFCache, so the industry doesn't seem to be well educated about it.  It's
>    probably on analytics providers to highlight these types of navigations
>    better.
>    - Michal: Similarly it is related to SPAs. SPAs that manually handle
>    the back button, won’t be counted as navigation as well.
>    - Yoav: So generally people don’t measure BF cache navigation just
>    yet, but it would be good to start?
>    - Gilles: How does one know it's a BFCache, is it in navigation.type?
>    - Yoav: It's actually not represented, the DOM is the same as from
>    before, i.e. navigation.type is the same as when the page first loaded.
>    You’re just getting a pageshow event
>    - Philip: Discussed at TPAC last year - Firefox were considering
>    updating the type in that case
>    - Yoav: There’s a “history navigation” type, but it’s not stated if
>    it’s a BF cache one or a regular history navigation. And in the case of BF
>    cache it’s not updated. We’d likely mint a new type.
>    - Michal: Important to track BF cache navigation, but the total number
>    of navigations a user commits will affect their metrics depending on how
>    you track it. Many things can influence that. User-behavior and how many
>    top-level navigations they commit during a session is important for
>    tracking the user experience.
>    - Yoav: UX can vary based on many factors. We’re trying to minimze
>    disruption by the BF cache introduction, but would be good to measure
>    things holistically.
>    - Michal: Comparison to Visibility is interesting. If the user chooses
>    to “open in new tab” and then go back, is that significantly different than
>    navigating to the new page and then clicking “back”?
>    - … We’re hoping that BF cache navigation is super fast, but so is
>    moving a tab from background to foreground. The overall perf story seems
>    very similar
>    - Yoav: So you want moving to BG and then FG to also count as a “fast
>    navigation”?
>    - Michal: If BF cache counts, so should this
>    - Yoav: BF cache is somewhat different, as if you’re not in the cache,
>    the experience is slower. But that’s also potentially true for BG, e.g. if
>    the browser threw your process out of memory, forcing a new load for the
>    page.
>    - Ilya: If I'm hearing you properly, there are many factors that
>    affect navigations.  In the particular context of BFCache, the industry is
>    currently ignoring that.
>    - …When we ship this in Chrome, it may look like performance has
>    regressed (because the denominator has changed), where the true story is
>    that it has improved.
>    - ... Because sites aren't looking at this dimension, they aren't
>    optimizing for it.  If you don't know that, you're excluding an easy win
>    for your site.
>    - Michal: Good summary. This is one of many examples where you can
>    influence the total number of navigations. I don’t see why this case is
>    special
>    - Yoav: Would be good to consider all of them holistically, and plan
>    for an extensible solution that can fit these future types.
>
> Next Call
>
>    - Proposal: Maybe worthwhile to move to Tuesday June 30th to avoid
>    holidays. Yoav to check.
>
>
> On Wed, Jun 17, 2020 at 8:37 AM Yoav Weiss <yoav@yoav.ws> wrote:
>
>> Hey folks!
>>
>> Join <https://meet.google.com/agz-fbji-spp> us this Thursday to talk
>> <https://docs.google.com/document/d/10dz_7QM5XCNsGeI63R864lF9gFqlqQD37B4q8Q46LMM/edit?pli=1#heading=h.1rw943dojw25>
>> about abandonment
>> <https://github.com/anniesullie/web-page-abandonment-api/blob/master/README.md>,
>> visibility
>> <https://docs.google.com/document/d/1A0u26wVip78yxlNrJdNKBQj3qdUrbWljY_NQgaIKab4/edit?ts=5ec4e44a> and
>> BF caches.
>>
>> As always the call will be recorded and posted online.
>>
>> See y'all there!!
>>
>> Cheers :)
>> Yoaav
>>
>>