Re: #282: Recommend minimum sizes for protocol elements

Hi Thomas,

On Sat, Jun 25, 2011 at 04:41:51AM +0000, Thomas Maslen wrote:
> Brian Pane <brianp@brianp.net> wrote:
> > On Friday, June 24, 2011, Mark Nottingham <mnot@mnot.net> wrote:
> >> Just to be clear -- we're talking about the total size of the *entire* header block here, not a single header limit.
> >>
> >> Were the folks arguing for 4k assuming the former or the latter?
> >
> > In my case, at least, the former: total size of the entire header block.
> 
> On the other end of the spectrum...
> 
> I work on server-side implementations (in various Java application servers) of HTTP Negotiate authentication (RFC 4559).
> 
> The "Authorization: Negotiate <Base64-blob>" headers that Windows clients send can be pretty big.  It is quite common for the HTTP header block to exceed 8 KB.  At one point I had convinced myself that 16 KB would always be comfortably large enough, but occasionally customers seem to run into trouble even then.
> 
> We are painfully aware of this because most of the HTTP implementations we deal with (various Java app servers, either with or without Apache HTTP Server front-ending them) default to an 8 KB limit on the size of the entire header block.  Thankfully, most of them provide a way to raise this limit, although a few do not (e.g. Jetty, IIRC).  Also, some of the connectors between Apache HTTP Server and Tomcat used to have a hard-coded protocol limit of 8 KB for the header block, although that has now been raised to 64 KB (16-bit length field in the protocol).
> 
> When an HTTP request hits the size limit, various servers respond in different ways, all of which lead to very puzzled customers.  Some servers send a 413 response (good).  Some just kill the connection and all you get to see is a TCP RST, which isn't very enlightening for debugging.  One app server did the nearest thing to a buffer overrun that you can do in Java.
> 
> 
> Now, perhaps HTTP Negotiate authentication is not a compelling example because it is mostly used in enterprise intranets, not out on the public internet.
> 
> However, I believe that limited HTTP header sizes are also problematic for some protocols on the public internet.  In particular, it seems to me that the protocols federated single sign-on (Liberty, WS-Federation, SAML) from browsers ended up pretty contorted because they needed to send a largish authentication object (a SAML token) that they couldn't reliably send in HTTP headers, so the protocols had to resort to hackery such as using JavaScript to make the browser client automatically issue an HTTP POST with the authentication object in the message-body.  (Yes, I realize there were other issues than header-size limits, particularly the requirement to work in existing browsers with no code changes).

You've described perfectly why recommending large values is the worst
thing to do : every time a large value is required, it is for a very
specific need, it causes a lot of trouble and in the end it can be
addressed correctly another way.

> So:  colour me very much in favour of the s/4k/20k/g that Mark floated.

That won't suddenly make the issues you encountered at 8k disappear. Quite
the opposite in fact, because instead of considering that anything above
4k requires double-checking, you'd blindly push your software everywhere
suggesting users that it will obviously cause no trouble since it's below
the 20k limit, but still their existing components will break as you
described above.

The recommended value is not a size limit. An application sending 100k can
be fine in some environments and still respect the rule. It's a bottom value
which any HTTP agent should always support, and that is safe to deal with.
As you showed it with your components, it's already not safe to bindly send
8k since it requires some specific configuration along the path.

Perhaps we could adapt the wording a bit (though I believe it was already
fine) to say something like :

   Various ad-hoc limitations on header length are found in practice. It is
   RECOMMENDED that all HTTP recipients support messages whose combined
   headers have 4,000 or more octets, so that it is always safe to send
   messages that large. Messages with larger headers might require specific
   configurations between the sender and the receiver and should not be
   considered as reliably deliverable without a preliminary check of the
   whole path.

Regards,
Willy

Received on Saturday, 25 June 2011 05:20:48 UTC