RE: HTTP 2.0 and a Faster, more Mobile-friendly web

Ø  Was the client/server speaking SPDY/2 or SPDY/3?
      Spdy/2


Ø  A percentage of the time, Chrome will not use SPDY (it decides this at startup). Did you verify that it was always speaking SPDY? (I'd guess yes!)

Yes.



Ø  Was the server using the CWND set sent in the SETTINGS frame, and was it sending it to the client?

I did not check this - I will go back and look.



Ø  What was the server's initcwnd?

Whatever the default is. I think it was 2 or 4. I will take a look.



Ø  Was slow-start-after-idle turned off on the kernel?

No. Again, we only used default settings.



Ø  How long were the various connections in slow start, and did they ever exit slow-start?

I will have to take a look.



Ø  Was the SSL cert chain the same size and using the same algorithms?

Yes.



Ø  Was the dummynet (or equivalent) configured to have appropriate buffer size (else it does packet drops silently)?

Yes, and I know this, because I had initially screwed this up :)



Ø  Were these all cold-connection, cold-cache first-page load timings? The use of keep-alive in figure 3 would seem to indicate otherwise, but I'm not sure which are which...

All cold cache. Keep alive is for only fetches while loading the first page.



Ø  Did you put Chromium into 'replay' mode (where date and random always return in the same manner, regardless of actual date)?

No, but I agree that this setting would have made more sense.



Ø  Any particular reason that you turned off entity-body compression in Apache? Were the resources saved verbatim from WinHTTrack (in which case it'd make sense)?

Not quite sure what you are asking here ... can you please clarify?



Ø  Was the server emulating the original server's think times? This can have a very, very large impact on pipelined HTTP performance.
      Only "index.html" was loaded for each site. I should clarify that in text. So think times are not an issue.


Ø  We've found that the various automatic page-is-done signals are often less useful than something which measures the above-the-fold page load time. This generally involves cameras pointed at screens and a human looking at the video to determine at which frame the page is useful, and designating that frame as 'done-for-above-the-fold'

I agree that age load time, as we measured it, is not a foolproof metric. The problem is that all other alternatives are too cumbersome to measure (and worse) replicate correctly.


Ø  I'd love to see if we could get a common suite of test data over which we could all run experiments/benchmarks. Is anyone else interested in doing so?

This is definitely interesting. Something like RFC5033?


From: Roberto Peon [mailto:grmocg@gmail.com]
Sent: Sunday, July 29, 2012 3:16 PM
To: Henrik Frystyk Nielsen
Cc: ietf-http-wg@w3.org; Gabriel Montenegro; Rob Trace; Adalberto Foresti (MS OPEN TECH)
Subject: Re: HTTP 2.0 and a Faster, more Mobile-friendly web


Interesting data! Questions about test methodology:

Was the client/server speaking SPDY/2 or SPDY/3?
A percentage of the time, Chrome will not use SPDY (it decides this at startup). Did you verify that it was always speaking SPDY? (I'd guess yes!)
Was the server using the CWND set sent in the SETTINGS frame, and was it sending it to the client?
What was the server's initcwnd?
Was slow-start-after-idle turned off on the kernel?
How long were the various connections in slow start, and did they ever exit slow-start?
Was the SSL cert chain the same size and using the same algorithms?
Was the dummynet (or equivalent) configured to have appropriate buffer size (else it does packet drops silently)?
Were these all cold-connection, cold-cache first-page load timings? The use of keep-alive in figure 3 would seem to indicate otherwise, but I'm not sure which are which...
Did you put Chromium into 'replay' mode (where date and random always return in the same manner, regardless of actual date)?
Any particular reason that you turned off entity-body compression in Apache? Were the resources saved verbatim from WinHTTrack (in which case it'd make sense)?
Was the server emulating the original server's think times? This can have a very, very large impact on pipelined HTTP performance.

We've found that the various automatic page-is-done signals are often less useful than something which measures the above-the-fold page load time. This generally involves cameras pointed at screens and a human looking at the video to determine at which frame the page is useful, and designating that frame as 'done-for-above-the-fold'

I'd love to see if we could get a common suite of test data over which we could all run experiments/benchmarks.
Is anyone else interested in doing so?

-=R
On Sun, Jul 29, 2012 at 1:14 PM, Henrik Frystyk Nielsen <henrikn@microsoft.com<mailto:henrikn@microsoft.com>> wrote:

Dear All,



We remain committed to the HTTP/2.0 standards process and look forward to seeing many of you this week at the IETF meeting in Vancouver to continue the discussion.  In the spirit of open discussion, we wanted to share some observations in advance of the meeting and share the latest progress from prototyping and testing.



There are currently three different proposals that the group is working through:



   * SPDY (http://tools.ietf.org/html/draft-mbelshe-httpbis-spdy),

   * HTTP Speed+Mobility (http://tools.ietf.org/html/draft-montenegro-httpbis-speed-mobility),

   * Network-Friendly HTTP Upgrade (http://tools.ietf.org/html/draft-tarreau-httpbis-network-friendly).



The good news is that everyone involved wants to make the Web faster, more scalable, more secure, and more mobile-friendly, and each proposal has benefits in different areas that the discussion can choose from.



--- A Genuinely Faster Web ---



The SPDY proposal has been great for raising awareness of Web performance. It takes a "clean slate" approach to improving HTTP.



To compare the performance of SPDY with HTTP/1.1 we have run tests comparing download times of several public web sites using a controlled tested study. The test uses publically available software run with mostly default configurations while applying all the currently available optimizations to HTTP/1.1. You can find a preliminary report on the test results here: http://research.microsoft.com/apps/pubs/?id=170059. The results mirror other data (http://www.guypo.com/technical/not-as-spdy-as-you-thought) that indicate mixed results with SPDY performance.



Our results indicate almost equal performance between SPDY and HTTP/1.1 when one applies all the known optimizations to HTTP/1.1. SPDY's performance improvements are not consistent and significant. We will continue our testing, and we welcome others to publish their results so that HTTP/2.0 can choose the best changes and deliver the best possible performance and scalability improvements compared to HTTP/1.1.





--- Taking the Best from Each ---



Speed is one of several areas of improvement. Currently, there's no clear consensus that any one of the proposals is the clear choice or even starting point for HTTP/2.0 (based on our reading the Expressions of Interest and discussions on this mailing list. A good example of this is the vigorous discussion around mandating TLS encryption (http://tools.ietf.org/html/rfc5246) for HTTP/2.0.



We think a good approach for HTTP/2.0 is to take the best solution for each of these areas from each of the proposals.  This approach helps us focus the discussion for each area of the protocol. Of course, this approach would still allow the standard to benefit from the extensive knowledge gained from implementing existing proposals.



We believe that the group can converge on consensus in the following areas, based on our reading of the Expressions of Interest, by starting from the different proposals.



------------------|------------------

Area              | Opinion that

                  | seems to prevail

------------------|------------------

1. Compression    | SPDY or Friendly

------------------|------------------

2. Multiplexing   | SPDY

------------------|------------------

3. Mandatory TLS  | Speed+Mobility

------------------|------------------

4. Negotiation    | Friendly or

                  |   Speed+Mobility

------------------|------------------

5. Client Pull/   | Speed+Mobility

      Server Push |

------------------|------------------

6. Flow Control   | SPDY

------------------|------------------

7. WebSockets     | Speed+Mobility

------------------|------------------



Below, we discuss each HTTP/2.0 element and the current consensus that appears to be forming within the Working Group.



1. Compression



Compression is simple to conceptualize and implement, and it is important. Proxies and other boxes in the middle on today's Web often face problems with it. The HTTP/2.0 discussion has been rich but with little consensus.

Though some studies suggest that SPDY's header compression approach shows promise, other studies show this compression to be prohibitively onerous for intermediary devices. More information here would help us make sure we're making the Web faster and better.



Also, an entire segment of implementers are not interested in compression as defined in SPDY.  That's a challenge because the latest strawman for the working group charter (http://lists.w3.org/Archives/Public/ietf-http-wg/2012JulSep/0784.html) states that the "resulting specification(s) are expected to be meet these goals for common existing deployments of HTTP; in particular, ... intermediation (by proxies, corporate firewalls, 'reverse' proxies and Content Delivery Networks)."



We think the SPDY or Friendly proposals is a good starting point for progress.



2. Multiplexing



All three proposals define similar multiplexing models. We haven't had substantial discussion on the differences. This lack of discussion suggests that there is rough consensus around the SPDY framing for multiplexing.



We think that the SPDY proposal is a good starting point here and best captures the current consensus.



3. Mandating Always On TLS



There is definitely no consensus to mandate TLS for all Web communication, but some major implementers have stated they will not to adopt HTTP/2.0 unless the working group supports a "TLS is mandatory" position. A very preliminary note from the chair (http://lists.w3.org/Archives/Public/ietf-http-wg/2012JulSep/0601.html) states that there is a lack of consensus for mandating TLS.



We think the Speed+Mobility proposal is a good starting point here as it provides options to turn TLS on (or not).



4. Negotiation



Only two of the proposals actually discuss how different endpoints agree to use HTTP/2.0.



(The SPDY proposal does not specify a negotiation method. Current prototype implementations use the TLS-NPN (http://tools.ietf.org/html/draft-agl-tls-nextprotoneg) extension.  While the other proposals use HTTP Upgrade to negotiate HTTP/2.0, some parties have expressed non-support for this method as well.)



We think either of the Friendly or Speed+Mobility proposals is a good starting point because they are the only ones that have any language in this respect.



5. Client Pull and Server Push



There are tradeoffs between a server push model and a client pull model. The main question is how to improve performance while respecting bandwidth and client caches.



Server Push has not had the same level of implementation and experimentation as the other features in SPDY. More information here would help us make sure we're making the Web faster and better.



We think the Speed+Mobility proposal is a good starting point here, suggesting that this issue may be better served in a separate document rather than tied to the core HTTP/2.0 protocol.



6. Flow Control



There has only been limited discussion in the HTTPbis working group on flow control. Flow Control offers a lot of opportunity make the Web faster as well as to break it; for example, implementations need to figure out how to optimize for opposing goals (like throughput and responsiveness) at the same time.



The current version of the SPDY proposal specifies a flow control message with many settings are that are not well-defined. The Speed+Mobilty proposal has a simplified flow control model based on certain assumptions. More experimentation and information here would help us make sure we're making the Web faster and better.



We think that the SPDY proposal is a good starting point here.



7. WebSockets



We see support  for aligning HTTP/2.0 with a future version of WebSockets, as suggested in the introduction of the Speed+Mobility proposal.





--- Moving forward ---



We're excited for the Web to get faster, more stable, and more capable, and HTTP/2.0 is an important part of that.



We believe that bringing together the best elements of the current SPDY, HTTP Speed+Mobility, and Network-Friendly HTTP Upgrade proposals is the best approach to make that happen.



Based on the discussions on the HTTPbis mailing list, we've suggested which proposals make the most sense to start from for each of the areas that HTTP/2.0 is addressing. Each of these areas needs more prototyping and experimentation and data. We're looking forward to the discussion this week.



Sincerely,



Henrik Frystyk Nielsen

Principal Architect, Microsoft Open Technologies, Inc.



Gabriel Montenegro

Principal Software Development Engineer, Microsoft Corporation



Rob Trace

Senior Program Manager Lead, Microsoft Corporation



Adalberto Foresti

Senior Program Manager, Microsoft Open Technologies, Inc.