- From: Tom Bergan <tombergan@chromium.org>
- Date: Mon, 22 Jul 2019 11:02:29 -0700
- To: "Nanner, Aman" <ananner@akamai.com>
- Cc: "ietf-http-wg@w3.org" <ietf-http-wg@w3.org>
- Message-ID: <CA+3+x5H9eF5mNGbLU_M3ZN=Jfe4PdstoeeZ-56G1UOZfRdOO4w@mail.gmail.com>
Hi Aman, Interesting results, thanks for sharing! I do wonder why you decided to compare means instead of percentiles? It looks like there are some crazy bad outliers. Assuming your "April 2019 Results with Old Methodology" and "April 2019 Results (1% quantile excluded)" graphs use the same set of sites in the same order, the mean goes from about +500ms to -200ms. If 99% of requests on that site take -200ms, the last 1% must take 70s! Instead of means, you could have compared medians, or 25th/75th percentiles, or other percentiles in the distribution. You can compute confidence intervals on percentiles. You can also do something like Mann-Whitney to check if the distributions differ significantly. I'd also like to hear more about this linear regression that you run on the A/B results. I couldn't follow why this regression is necessary. If your A/B test uses i.i.d. coin flips, you can compute confidence intervals directly on the A/B results. Is your A/B test not i.i.d.? How do you know the linear regression doesn't introduce bias? Also, have you looked into measuring other metrics, such as FirstContentfulPaint? -Tom On Mon, Jul 22, 2019 at 4:41 AM Nanner, Aman <ananner@akamai.com> wrote: > At the previous IETF 102 HTTPWG session in Montreal, I had presented some > Akamai data on H2 Server Push which can be found here: > > > https://github.com/httpwg/wg-materials/blob/gh-pages/ietf102/akamai-server-push.pdf > > Akamai has conducted some more recent tests with a tweaked methodology > (exclusion of 1% highest-latency requests on the long-tail), and we have > found some interesting results. I share some more details about the > performance analysis here: > > > https://medium.com/@ananner/http-2-server-push-performance-a-further-akamai-case-study-7a17573a3317 > > Thanks, > Aman Nanner > Akamai Technologies Inc. > >
Received on Monday, 22 July 2019 18:03:39 UTC