Re: H2 Server Push performance data

Did you at least do an A/A comparison to test if your regression introduces
bias?

On Mon, Jul 22, 2019 at 2:27 PM Nanner, Aman <ananner@akamai.com> wrote:

> Hi Tom,
>
> Thanks for your feedback!
>
> The first couple of charts (based on the multiple customer sites) use the
> mean as a way to provide a summary view of performance for each individual
> site.  The last chart in the article is a percentile plot for one customer
> site which demonstrates a more granular view, but we do not have percentile
> plots put together for each customer site. It should be possible to do that
> in the event we were to perform deeper analysis on more sites.
>
> The A/B data was not collected in a way that we would consider to be pure
> enough to be considered i.i.d coin flips. There exist dimensions where the
> values can make it more or less likely that Server Push may be applied on
> specific responses within the production network (e.g. geo, time of day,
> user-agent), and so the approach was chosen to use linear regression to
> attempt to control for these variables.
>
> I agree that the FirstContentfulPaint metric would be useful to analyze;
> we had some limitations in our dataset regarding collection of that metric
> that resulted in it not being reliable enough to use (it wasn't as widely
> collected as DOMComplete). There have been some encouraging results seen
> for this metric based on some preliminary analysis, but we wouldn't
> consider the data and analysis reliable enough yet for sharing.
>
> Thanks,
> Aman
>
>
> ------------------------------
> *From:* Tom Bergan <tombergan@chromium.org>
> *Sent:* Monday, July 22, 2019 2:02 PM
> *To:* Nanner, Aman
> *Cc:* ietf-http-wg@w3.org
> *Subject:* Re: H2 Server Push performance data
>
> Hi Aman,
>
> Interesting results, thanks for sharing! I do wonder why you decided to
> compare means instead of percentiles? It looks like there are some crazy
> bad outliers. Assuming your "April 2019 Results with Old Methodology" and
> "April 2019 Results (1% quantile excluded)" graphs use the same set of
> sites in the same order, the mean goes from about +500ms to -200ms. If 99%
> of requests on that site take -200ms, the last 1% must take 70s! Instead of
> means, you could have compared medians, or 25th/75th percentiles, or other
> percentiles in the distribution. You can compute confidence intervals on
> percentiles. You can also do something like Mann-Whitney to check if the
> distributions differ significantly.
>
> I'd also like to hear more about this linear regression that you run on
> the A/B results. I couldn't follow why this regression is necessary. If
> your A/B test uses i.i.d. coin flips, you can compute confidence intervals
> directly on the A/B results. Is your A/B test not i.i.d.? How do you know
> the linear regression doesn't introduce bias?
>
> Also, have you looked into measuring other metrics, such as
> FirstContentfulPaint?
>
> -Tom
>
> On Mon, Jul 22, 2019 at 4:41 AM Nanner, Aman <ananner@akamai.com> wrote:
>
>> At the previous IETF 102 HTTPWG session in Montreal, I had presented some
>> Akamai data on H2 Server Push which can be found here:
>>
>>
>> https://github.com/httpwg/wg-materials/blob/gh-pages/ietf102/akamai-server-push.pdf
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_httpwg_wg-2Dmaterials_blob_gh-2Dpages_ietf102_akamai-2Dserver-2Dpush.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=HVAbQ4bnkmcYKMR6_H0nEMfpVIpY1jgJWVBhC_es5Nw&m=JhnrDdJ2FDK0-S_OawS0GoKyXIVn4C8H_c03AveZDRo&s=UzEaCDDavuWPtu3zi_hdJa51YwKMwdzescfn9rpYnlQ&e=>
>>
>> Akamai has conducted some more recent tests with a tweaked methodology
>> (exclusion of 1% highest-latency requests on the long-tail), and we have
>> found some interesting results. I share some more details about the
>> performance analysis here:
>>
>>
>> https://medium.com/@ananner/http-2-server-push-performance-a-further-akamai-case-study-7a17573a3317
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__medium.com_-40ananner_http-2D2-2Dserver-2Dpush-2Dperformance-2Da-2Dfurther-2Dakamai-2Dcase-2Dstudy-2D7a17573a3317&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=HVAbQ4bnkmcYKMR6_H0nEMfpVIpY1jgJWVBhC_es5Nw&m=JhnrDdJ2FDK0-S_OawS0GoKyXIVn4C8H_c03AveZDRo&s=5ScQ7t8fvPbCcHySFH1WDh9iHpnjchASXvAmdo8mvJM&e=>
>>
>> Thanks,
>> Aman Nanner
>> Akamai Technologies Inc.
>>
>>