Re: HTTP 2.0 and a Faster, more Mobile-friendly web

Interesting data! Questions about test methodology:

Was the client/server speaking SPDY/2 or SPDY/3?
A percentage of the time, Chrome will not use SPDY (it decides this at
startup). Did you verify that it was always speaking SPDY? (I'd guess yes!)
Was the server using the CWND set sent in the SETTINGS frame, and was it
sending it to the client?
What was the server's initcwnd?
Was slow-start-after-idle turned off on the kernel?
How long were the various connections in slow start, and did they ever exit
slow-start?
Was the SSL cert chain the same size and using the same algorithms?
Was the dummynet (or equivalent) configured to have appropriate buffer size
(else it does packet drops silently)?
Were these all cold-connection, cold-cache first-page load timings? The use
of keep-alive in figure 3 would seem to indicate otherwise, but I'm not
sure which are which...
Did you put Chromium into 'replay' mode (where date and random always
return in the same manner, regardless of actual date)?
Any particular reason that you turned off entity-body compression in
Apache? Were the resources saved verbatim from WinHTTrack (in which case
it'd make sense)?
Was the server emulating the original server's think times? This can have a
very, very large impact on pipelined HTTP performance.

We've found that the various automatic page-is-done signals are often less
useful than something which measures the above-the-fold page load time.
This generally involves cameras pointed at screens and a human looking at
the video to determine at which frame the page is useful, and designating
that frame as 'done-for-above-the-fold'

I'd love to see if we could get a common suite of test data over which we
could all run experiments/benchmarks.
Is anyone else interested in doing so?

-=R

On Sun, Jul 29, 2012 at 1:14 PM, Henrik Frystyk Nielsen <
henrikn@microsoft.com> wrote:

>  Dear All,****
>
> ** **
>
> We remain committed to the HTTP/2.0 standards process and look forward to
> seeing many of you this week at the IETF meeting in Vancouver to continue
> the discussion.  In the spirit of open discussion, we wanted to share some
> observations in advance of the meeting and share the latest progress from
> prototyping and testing. ****
>
> ** **
>
> There are currently three different proposals that the group is working
> through:****
>
> ** **
>
>    * SPDY (http://tools.ietf.org/html/draft-mbelshe-httpbis-spdy),****
>
>    * HTTP Speed+Mobility (
> http://tools.ietf.org/html/draft-montenegro-httpbis-speed-mobility),****
>
>    * Network-Friendly HTTP Upgrade (
> http://tools.ietf.org/html/draft-tarreau-httpbis-network-friendly). ****
>
> ** **
>
> The good news is that everyone involved wants to make the Web faster, more
> scalable, more secure, and more mobile-friendly, and each proposal has
> benefits in different areas that the discussion can choose from.****
>
> ** **
>
> --- A Genuinely Faster Web ---****
>
> ** **
>
> The SPDY proposal has been great for raising awareness of Web performance.
> It takes a “clean slate” approach to improving HTTP.****
>
> ** **
>
> To compare the performance of SPDY with HTTP/1.1 we have run tests
> comparing download times of several public web sites using a controlled
> tested study. The test uses publically available software run with mostly
> default configurations while applying all the currently available
> optimizations to HTTP/1.1. You can find a preliminary report on the test
> results here: http://research.microsoft.com/apps/pubs/?id=170059. The
> results mirror other data (
> http://www.guypo.com/technical/not-as-spdy-as-you-thought) that indicate
> mixed results with SPDY performance.****
>
> ** **
>
> Our results indicate almost equal performance between SPDY and HTTP/1.1
> when one applies all the known optimizations to HTTP/1.1. SPDY’s
> performance improvements are not consistent and significant. We will
> continue our testing, and we welcome others to publish their results so
> that HTTP/2.0 can choose the best changes and deliver the best possible
> performance and scalability improvements compared to HTTP/1.1.****
>
> ** **
>
> ** **
>
> --- Taking the Best from Each ---****
>
> ** **
>
> Speed is one of several areas of improvement. Currently, there’s no clear
> consensus that any one of the proposals is the clear choice or even
> starting point for HTTP/2.0 (based on our reading the Expressions of
> Interest and discussions on this mailing list. A good example of this is
> the vigorous discussion around mandating TLS encryption (
> http://tools.ietf.org/html/rfc5246) for HTTP/2.0. ****
>
> ** **
>
> We think a good approach for HTTP/2.0 is to take the best solution for
> each of these areas from each of the proposals.  This approach helps us
> focus the discussion for each area of the protocol. Of course, this
> approach would still allow the standard to benefit from the extensive
> knowledge gained from implementing existing proposals.  ****
>
> ** **
>
> We believe that the group can converge on consensus in the following
> areas, based on our reading of the Expressions of Interest, by starting
> from the different proposals.****
>
> ** **
>
> ------------------|------------------****
>
> Area              | Opinion that ****
>
>                   | seems to prevail****
>
> ------------------|------------------****
>
> 1. Compression    | SPDY or Friendly****
>
> ------------------|------------------****
>
> 2. Multiplexing   | SPDY****
>
> ------------------|------------------****
>
> 3. Mandatory TLS  | Speed+Mobility****
>
> ------------------|------------------****
>
> 4. Negotiation    | Friendly or****
>
>                   |   Speed+Mobility****
>
> ------------------|------------------****
>
> 5. Client Pull/   | Speed+Mobility****
>
>       Server Push | ****
>
> ------------------|------------------****
>
> 6. Flow Control   | SPDY****
>
> ------------------|------------------****
>
> 7. WebSockets     | Speed+Mobility****
>
> ------------------|------------------****
>
> ** **
>
> Below, we discuss each HTTP/2.0 element and the current consensus that
> appears to be forming within the Working Group.****
>
> ** **
>
> 1. Compression****
>
> ** **
>
> Compression is simple to conceptualize and implement, and it is important.
> Proxies and other boxes in the middle on today’s Web often face problems
> with it. The HTTP/2.0 discussion has been rich but with little consensus.*
> ***
>
> ****
>
> Though some studies suggest that SPDY’s header compression approach shows
> promise, other studies show this compression to be prohibitively onerous
> for intermediary devices. More information here would help us make sure
> we’re making the Web faster and better. ****
>
> ** **
>
> Also, an entire segment of implementers are not interested in compression
> as defined in SPDY.  That’s a challenge because the latest strawman for the
> working group charter (
> http://lists.w3.org/Archives/Public/ietf-http-wg/2012JulSep/0784.html)
> states that the “resulting specification(s) are expected to be meet these
> goals for common existing deployments of HTTP; in particular, …
> intermediation (by proxies, corporate firewalls, ‘reverse’ proxies and
> Content Delivery Networks).” ****
>
> ** **
>
> We think the SPDY or Friendly proposals is a good starting point for
> progress.****
>
> ** **
>
> 2. Multiplexing****
>
> ** **
>
> All three proposals define similar multiplexing models. We haven’t had
> substantial discussion on the differences. This lack of discussion suggests
> that there is rough consensus around the SPDY framing for multiplexing. **
> **
>
> ** **
>
> We think that the SPDY proposal is a good starting point here and best
> captures the current consensus.****
>
> ** **
>
> 3. Mandating Always On TLS ****
>
> ** **
>
> There is definitely no consensus to mandate TLS for all Web communication,
> but some major implementers have stated they will not to adopt HTTP/2.0
> unless the working group supports a “TLS is mandatory” position. A very
> preliminary note from the chair (
> http://lists.w3.org/Archives/Public/ietf-http-wg/2012JulSep/0601.html)
> states that there is a lack of consensus for mandating TLS. ****
>
> ** **
>
> We think the Speed+Mobility proposal is a good starting point here as it
> provides options to turn TLS on (or not).****
>
> ** **
>
> 4. Negotiation****
>
> ** **
>
> Only two of the proposals actually discuss how different endpoints agree
> to use HTTP/2.0. ****
>
> ** **
>
> (The SPDY proposal does not specify a negotiation method. Current
> prototype implementations use the TLS-NPN (
> http://tools.ietf.org/html/draft-agl-tls-nextprotoneg) extension.  While
> the other proposals use HTTP Upgrade to negotiate HTTP/2.0, some parties
> have expressed non-support for this method as well.) ****
>
> ** **
>
> We think either of the Friendly or Speed+Mobility proposals is a good
> starting point because they are the only ones that have any language in
> this respect.  ****
>
> ** **
>
> 5. Client Pull and Server Push****
>
> ** **
>
> There are tradeoffs between a server push model and a client pull model.
> The main question is how to improve performance while respecting bandwidth
> and client caches. ****
>
> ** **
>
> Server Push has not had the same level of implementation and
> experimentation as the other features in SPDY. More information here would
> help us make sure we’re making the Web faster and better. ****
>
> ** **
>
> We think the Speed+Mobility proposal is a good starting point here,
> suggesting that this issue may be better served in a separate document
> rather than tied to the core HTTP/2.0 protocol. ****
>
> ** **
>
> 6. Flow Control****
>
> ** **
>
> There has only been limited discussion in the HTTPbis working group on
> flow control. Flow Control offers a lot of opportunity make the Web faster
> as well as to break it; for example, implementations need to figure out how
> to optimize for opposing goals (like throughput and responsiveness) at the
> same time. ****
>
> ** **
>
> The current version of the SPDY proposal specifies a flow control message
> with many settings are that are not well-defined. The Speed+Mobilty
> proposal has a simplified flow control model based on certain assumptions.
> More experimentation and information here would help us make sure we’re
> making the Web faster and better. ****
>
> ** **
>
> We think that the SPDY proposal is a good starting point here. ****
>
> ** **
>
> 7. WebSockets****
>
> ** **
>
> We see support  for aligning HTTP/2.0 with a future version of WebSockets,
> as suggested in the introduction of the Speed+Mobility proposal.****
>
> ** **
>
> ** **
>
> --- Moving forward ---****
>
> ** **
>
> We’re excited for the Web to get faster, more stable, and more capable,
> and HTTP/2.0 is an important part of that. ****
>
> ** **
>
> We believe that bringing together the best elements of the current SPDY,
> HTTP Speed+Mobility, and Network-Friendly HTTP Upgrade proposals is the
> best approach to make that happen. ****
>
> ** **
>
> Based on the discussions on the HTTPbis mailing list, we’ve suggested
> which proposals make the most sense to start from for each of the areas
> that HTTP/2.0 is addressing. Each of these areas needs more prototyping and
> experimentation and data. We’re looking forward to the discussion this week.
> ****
>
> ** **
>
> Sincerely,****
>
> ** **
>
> Henrik Frystyk Nielsen****
>
> Principal Architect, Microsoft Open Technologies, Inc.****
>
> ** **
>
> Gabriel Montenegro****
>
> Principal Software Development Engineer, Microsoft Corporation****
>
> ** **
>
> Rob Trace****
>
> Senior Program Manager Lead, Microsoft Corporation****
>
> ** **
>
> Adalberto Foresti****
>
> Senior Program Manager, Microsoft Open Technologies, Inc.****
>
> ** **
>
> ** **
>