Re: HTTP 2.0 and a Faster, more Mobile-friendly web

Thanks for answering by questions!

On Mon, Jul 30, 2012 at 10:36 AM, Jitu Padhye <padhye@microsoft.com> wrote:

>  **Ø  **Was the client/server speaking SPDY/2 or SPDY/3?****
>
>       Spdy/2****
>
> ** **
>
> **Ø  **A percentage of the time, Chrome will not use SPDY (it decides
> this at startup). Did you verify that it was always speaking SPDY? (I'd
> guess yes!)****
>
> Yes. ****
>
> ** **
>
> **Ø  **Was the server using the CWND set sent in the SETTINGS frame, and
> was it sending it to the client?****
>
> I did not check this – I will go back and look. ** **
>
> ** **
>
> **Ø  **What was the server's initcwnd?****
>
> Whatever the default is. I think it was 2 or 4. I will take a look.
>

A default of 10 is suggested. (This is the case for newer linux kernels by
default).

> ****
>
> ** **
>
> **Ø  **Was slow-start-after-idle turned off on the kernel?****
>
> No. Again, we only used default settings. ****
>
> ** **
>
> **Ø  **How long were the various connections in slow start, and did they
> ever exit slow-start?****
>
> I will have to take a look. ****
>
> ** **
>
> **Ø  **Was the SSL cert chain the same size and using the same algorithms?
> ****
>
> Yes. ****
>
> ** **
>
> **Ø  **Was the dummynet (or equivalent) configured to have appropriate
> buffer size (else it does packet drops silently)?****
>
> Yes, and I know this, because I had initially screwed this up J
>

Yea, it is easy to mess this one up-- good catch!


> ****
>
> ** **
>
> **Ø  **Were these all cold-connection, cold-cache first-page load
> timings? The use of keep-alive in figure 3 would seem to indicate
> otherwise, but I'm not sure which are which...****
>
> All cold cache. Keep alive is for only fetches while loading the first
> page.
>
It would be instructive to look at more than just the first page load. I
suspect most site designers and sites intend to have the user continue
interacting with the site for more than just the one page view :)

This is the curse of the benchmark-- someone always demanding more. In this
case, I do think it is important, though, since the pattern is further site
involvement much of the time.

> ****
>
> ** **
>
> **Ø  **Did you put Chromium into 'replay' mode (where date and random
> always return in the same manner, regardless of actual date)?****
>
> No, but I agree that this setting would have made more sense. ****
>
> ** **
>
> **Ø  **Any particular reason that you turned off entity-body compression
> in Apache? Were the resources saved verbatim from WinHTTrack (in which case
> it'd make sense)?****
>
> Not quite sure what you are asking here … can you please clarify?
>

I think this question may not make sense if resources other than index.html
were not served from the server as I think you're saying elsewhere?
What I mean was: If the entity-body for the site was originally compressed,
are you sending it compressed in your tests, or would it end up
uncompressed?


> ****
>
> ** **
>
> **Ø  **Was the server emulating the original server's think times? This
> can have a very, very large impact on pipelined HTTP performance. ****
>
>       Only “index.html” was loaded for each site. I should clarify that in
> text. So think times are not an issue.
>

The resources referenced from index.html are, or are not part of the test?
I'd guess that they are, and I've confused you somehow.


> ****
>
> ** **
>
> **Ø  **We've found that the various automatic page-is-done signals are
> often less useful than something which measures the above-the-fold page
> load time. This generally involves cameras pointed at screens and a human
> looking at the video to determine at which frame the page is useful, and
> designating that frame as 'done-for-above-the-fold'****
>
> I agree that age load time, as we measured it, is not a foolproof metric.
> The problem is that all other alternatives are too cumbersome to measure
> (and worse) replicate correctly. ****
>
> ** **
>
> **Ø  **I'd love to see if we could get a common suite of test data over
> which we could all run experiments/benchmarks. Is anyone else interested in
> doing so?****
>
> This is definitely interesting. Something like RFC5033?
>

I'm not finding what I'm suggesting in that RFC, but it is possible it is
there :)

All I mean is that we grab a number of pages that represent important
use-cases, then, as necessary, clean up the resources so that there are no
legal issues w.r.t. distribution, and then have this used as a basis for
benchmarking and testing of corner-cases.
-=R


> **
>
> ** **
>
> ** **
>
> *From:* Roberto Peon [mailto:grmocg@gmail.com]
> *Sent:* Sunday, July 29, 2012 3:16 PM
> *To:* Henrik Frystyk Nielsen
> *Cc:* ietf-http-wg@w3.org; Gabriel Montenegro; Rob Trace; Adalberto
> Foresti (MS OPEN TECH)
>
> *Subject:* Re: HTTP 2.0 and a Faster, more Mobile-friendly web****
>
> ** **
>
> ** **
>
> Interesting data! Questions about test methodology:****
>
> ** **
>
> Was the client/server speaking SPDY/2 or SPDY/3?****
>
> A percentage of the time, Chrome will not use SPDY (it decides this at
> startup). Did you verify that it was always speaking SPDY? (I'd guess yes!)
> ****
>
> Was the server using the CWND set sent in the SETTINGS frame, and was it
> sending it to the client?****
>
> What was the server's initcwnd?****
>
> Was slow-start-after-idle turned off on the kernel?****
>
> How long were the various connections in slow start, and did they ever
> exit slow-start?****
>
> Was the SSL cert chain the same size and using the same algorithms?****
>
> Was the dummynet (or equivalent) configured to have appropriate buffer
> size (else it does packet drops silently)?****
>
> Were these all cold-connection, cold-cache first-page load timings? The
> use of keep-alive in figure 3 would seem to indicate otherwise, but I'm not
> sure which are which...****
>
> Did you put Chromium into 'replay' mode (where date and random always
> return in the same manner, regardless of actual date)?****
>
> Any particular reason that you turned off entity-body compression in
> Apache? Were the resources saved verbatim from WinHTTrack (in which case
> it'd make sense)?****
>
> Was the server emulating the original server's think times? This can have
> a very, very large impact on pipelined HTTP performance. ****
>
> ** **
>
> We've found that the various automatic page-is-done signals are often less
> useful than something which measures the above-the-fold page load time.
> This generally involves cameras pointed at screens and a human looking at
> the video to determine at which frame the page is useful, and designating
> that frame as 'done-for-above-the-fold'****
>
> ** **
>
> I'd love to see if we could get a common suite of test data over which we
> could all run experiments/benchmarks.****
>
> Is anyone else interested in doing so?****
>
> ** **
>
> -=R****
>
> On Sun, Jul 29, 2012 at 1:14 PM, Henrik Frystyk Nielsen <
> henrikn@microsoft.com> wrote:****
>
> Dear All,****
>
>  ****
>
> We remain committed to the HTTP/2.0 standards process and look forward to
> seeing many of you this week at the IETF meeting in Vancouver to continue
> the discussion.  In the spirit of open discussion, we wanted to share some
> observations in advance of the meeting and share the latest progress from
> prototyping and testing. ****
>
>  ****
>
> There are currently three different proposals that the group is working
> through:****
>
>  ****
>
>    * SPDY (http://tools.ietf.org/html/draft-mbelshe-httpbis-spdy),****
>
>    * HTTP Speed+Mobility (
> http://tools.ietf.org/html/draft-montenegro-httpbis-speed-mobility),****
>
>    * Network-Friendly HTTP Upgrade (
> http://tools.ietf.org/html/draft-tarreau-httpbis-network-friendly). ****
>
>  ****
>
> The good news is that everyone involved wants to make the Web faster, more
> scalable, more secure, and more mobile-friendly, and each proposal has
> benefits in different areas that the discussion can choose from.****
>
>  ****
>
> --- A Genuinely Faster Web ---****
>
>  ****
>
> The SPDY proposal has been great for raising awareness of Web performance.
> It takes a “clean slate” approach to improving HTTP.****
>
>  ****
>
> To compare the performance of SPDY with HTTP/1.1 we have run tests
> comparing download times of several public web sites using a controlled
> tested study. The test uses publically available software run with mostly
> default configurations while applying all the currently available
> optimizations to HTTP/1.1. You can find a preliminary report on the test
> results here: http://research.microsoft.com/apps/pubs/?id=170059. The
> results mirror other data (
> http://www.guypo.com/technical/not-as-spdy-as-you-thought) that indicate
> mixed results with SPDY performance.****
>
>  ****
>
> Our results indicate almost equal performance between SPDY and HTTP/1.1
> when one applies all the known optimizations to HTTP/1.1. SPDY’s
> performance improvements are not consistent and significant. We will
> continue our testing, and we welcome others to publish their results so
> that HTTP/2.0 can choose the best changes and deliver the best possible
> performance and scalability improvements compared to HTTP/1.1.****
>
>  ****
>
>  ****
>
> --- Taking the Best from Each ---****
>
>  ****
>
> Speed is one of several areas of improvement. Currently, there’s no clear
> consensus that any one of the proposals is the clear choice or even
> starting point for HTTP/2.0 (based on our reading the Expressions of
> Interest and discussions on this mailing list. A good example of this is
> the vigorous discussion around mandating TLS encryption (
> http://tools.ietf.org/html/rfc5246) for HTTP/2.0. ****
>
>  ****
>
> We think a good approach for HTTP/2.0 is to take the best solution for
> each of these areas from each of the proposals.  This approach helps us
> focus the discussion for each area of the protocol. Of course, this
> approach would still allow the standard to benefit from the extensive
> knowledge gained from implementing existing proposals.  ****
>
>  ****
>
> We believe that the group can converge on consensus in the following
> areas, based on our reading of the Expressions of Interest, by starting
> from the different proposals.****
>
>  ****
>
> ------------------|------------------****
>
> Area              | Opinion that ****
>
>                   | seems to prevail****
>
> ------------------|------------------****
>
> 1. Compression    | SPDY or Friendly****
>
> ------------------|------------------****
>
> 2. Multiplexing   | SPDY****
>
> ------------------|------------------****
>
> 3. Mandatory TLS  | Speed+Mobility****
>
> ------------------|------------------****
>
> 4. Negotiation    | Friendly or****
>
>                   |   Speed+Mobility****
>
> ------------------|------------------****
>
> 5. Client Pull/   | Speed+Mobility****
>
>       Server Push | ****
>
> ------------------|------------------****
>
> 6. Flow Control   | SPDY****
>
> ------------------|------------------****
>
> 7. WebSockets     | Speed+Mobility****
>
> ------------------|------------------****
>
>  ****
>
> Below, we discuss each HTTP/2.0 element and the current consensus that
> appears to be forming within the Working Group.****
>
>  ****
>
> 1. Compression****
>
>  ****
>
> Compression is simple to conceptualize and implement, and it is important.
> Proxies and other boxes in the middle on today’s Web often face problems
> with it. The HTTP/2.0 discussion has been rich but with little consensus.*
> ***
>
> Though some studies suggest that SPDY’s header compression approach shows
> promise, other studies show this compression to be prohibitively onerous
> for intermediary devices. More information here would help us make sure
> we’re making the Web faster and better. ****
>
>  ****
>
> Also, an entire segment of implementers are not interested in compression
> as defined in SPDY.  That’s a challenge because the latest strawman for the
> working group charter (
> http://lists.w3.org/Archives/Public/ietf-http-wg/2012JulSep/0784.html)
> states that the “resulting specification(s) are expected to be meet these
> goals for common existing deployments of HTTP; in particular, …
> intermediation (by proxies, corporate firewalls, ‘reverse’ proxies and
> Content Delivery Networks).” ****
>
>  ****
>
> We think the SPDY or Friendly proposals is a good starting point for
> progress.****
>
>  ****
>
> 2. Multiplexing****
>
>  ****
>
> All three proposals define similar multiplexing models. We haven’t had
> substantial discussion on the differences. This lack of discussion suggests
> that there is rough consensus around the SPDY framing for multiplexing. **
> **
>
>  ****
>
> We think that the SPDY proposal is a good starting point here and best
> captures the current consensus.****
>
>  ****
>
> 3. Mandating Always On TLS ****
>
>  ****
>
> There is definitely no consensus to mandate TLS for all Web communication,
> but some major implementers have stated they will not to adopt HTTP/2.0
> unless the working group supports a “TLS is mandatory” position. A very
> preliminary note from the chair (
> http://lists.w3.org/Archives/Public/ietf-http-wg/2012JulSep/0601.html)
> states that there is a lack of consensus for mandating TLS. ****
>
>  ****
>
> We think the Speed+Mobility proposal is a good starting point here as it
> provides options to turn TLS on (or not).****
>
>  ****
>
> 4. Negotiation****
>
>  ****
>
> Only two of the proposals actually discuss how different endpoints agree
> to use HTTP/2.0. ****
>
>  ****
>
> (The SPDY proposal does not specify a negotiation method. Current
> prototype implementations use the TLS-NPN (
> http://tools.ietf.org/html/draft-agl-tls-nextprotoneg) extension.  While
> the other proposals use HTTP Upgrade to negotiate HTTP/2.0, some parties
> have expressed non-support for this method as well.) ****
>
>  ****
>
> We think either of the Friendly or Speed+Mobility proposals is a good
> starting point because they are the only ones that have any language in
> this respect.  ****
>
>  ****
>
> 5. Client Pull and Server Push****
>
>  ****
>
> There are tradeoffs between a server push model and a client pull model.
> The main question is how to improve performance while respecting bandwidth
> and client caches. ****
>
>  ****
>
> Server Push has not had the same level of implementation and
> experimentation as the other features in SPDY. More information here would
> help us make sure we’re making the Web faster and better. ****
>
>  ****
>
> We think the Speed+Mobility proposal is a good starting point here,
> suggesting that this issue may be better served in a separate document
> rather than tied to the core HTTP/2.0 protocol. ****
>
>  ****
>
> 6. Flow Control****
>
>  ****
>
> There has only been limited discussion in the HTTPbis working group on
> flow control. Flow Control offers a lot of opportunity make the Web faster
> as well as to break it; for example, implementations need to figure out how
> to optimize for opposing goals (like throughput and responsiveness) at the
> same time. ****
>
>  ****
>
> The current version of the SPDY proposal specifies a flow control message
> with many settings are that are not well-defined. The Speed+Mobilty
> proposal has a simplified flow control model based on certain assumptions.
> More experimentation and information here would help us make sure we’re
> making the Web faster and better. ****
>
>  ****
>
> We think that the SPDY proposal is a good starting point here. ****
>
>  ****
>
> 7. WebSockets****
>
>  ****
>
> We see support  for aligning HTTP/2.0 with a future version of WebSockets,
> as suggested in the introduction of the Speed+Mobility proposal.****
>
>  ****
>
>  ****
>
> --- Moving forward ---****
>
>  ****
>
> We’re excited for the Web to get faster, more stable, and more capable,
> and HTTP/2.0 is an important part of that. ****
>
>  ****
>
> We believe that bringing together the best elements of the current SPDY,
> HTTP Speed+Mobility, and Network-Friendly HTTP Upgrade proposals is the
> best approach to make that happen. ****
>
>  ****
>
> Based on the discussions on the HTTPbis mailing list, we’ve suggested
> which proposals make the most sense to start from for each of the areas
> that HTTP/2.0 is addressing. Each of these areas needs more prototyping and
> experimentation and data. We’re looking forward to the discussion this week.
> ****
>
>  ****
>
> Sincerely,****
>
>  ****
>
> Henrik Frystyk Nielsen****
>
> Principal Architect, Microsoft Open Technologies, Inc.****
>
>  ****
>
> Gabriel Montenegro****
>
> Principal Software Development Engineer, Microsoft Corporation****
>
>  ****
>
> Rob Trace****
>
> Senior Program Manager Lead, Microsoft Corporation****
>
>  ****
>
> Adalberto Foresti****
>
> Senior Program Manager, Microsoft Open Technologies, Inc.****
>
>  ****
>
>  ****
>
> ** **
>