Re: Large Frame Proposal

On Mon, Jul 7, 2014 at 12:30 PM, Willy Tarreau <w@1wt.eu> wrote:

> On Mon, Jul 07, 2014 at 11:48:04AM -0700, Nicholas Hurley wrote:
> > On Mon, Jul 7, 2014 at 11:17 AM, Willy Tarreau <w@1wt.eu> wrote:
> >
> > > For example, the 0.02% you're talking about are the ones that *require*
> > > support for CONTINUATION in *all* implementations and which make all
> > > these implementations prone to slowloris-like attacks (for example).
> > >
> >
> > There will always be attacks. We take one away, another will be created.
> We
> > can mitigate this one without changing the entire framing of the
> protocol.
> > For example, moving END_STREAM to CONTINUATION instead of HEADERS is a
> very
> > quick, easy indicator of whether or not the header block has more frames
> > coming. That makes it easier to decide whether or not to kill the
> > connection after the HEADERS frame.
>
> Here it's irrelevant to this kind of attacks. Currently nothing prevents
> a user-agent from sending headers one byte at a time using CONT frames.
> Someone suggested that even 0 bytes might work as well. Quite an easy way
> to consume all of a servers' resources with a very low bandwidth. It even
> amplifies what we used to require over TCP because here you can even open
> many streams inside a single TCP connection using a single TCP segment.
>

That is true even with jumbo frames.
You send a large frame size, then send one byte and wait for a while, then
send another byte...
This pattern is a classic attack vector regardless of the framing
mechanism-- one must detect for it and handle it regardless.
Arguably, the jumbo-frame thing suffers more from this as, in order to get
CPU efficiencies, one allocates a large memory buffer for this instead of
resizing/partitioning, and thus commits a larger fraction of memory to the
attacker(s) performing the slowloris attack.


>
> The issue is that due to the way the protocol is designed right now, there
> is an easy incentive for not sending headers in one block, and poor designs
> will probably do so, requiring servers to always support CONTINUATIONS and
> be prone to such weaknesses. Not to mention the burden of dealing with
> them.


> > We could also make a change to indicate
> > the total size of the header block before sending anything, which also
> > mitigates the slowloris-style attack (at the expense of not being able to
> > stream HPACK - a tradeoff I am thinking more and more that I would be
> > willing to make).
>
> It could be better, but will not completely defeat such attacks, as you can
> easily advertise 16kB of headers and send 16383 blocks of 1 byte.
>

As mentioned above, an attacker can do the same thing with the jumbo frame.


>
> > I'm also still not convinced by the state commitment issue. It's entirely
> > possible for an endpoint to process and discard header data (thus
> > maintaining the shared state for the connection) without having to hold
> on
> > to the resulting headers. This means that endpoint is only keeping the
> > state that it has already committed to keeping (namely, the shared HPACK
> > state). So, it turns out, I was wrong, you don't have to send a GOAWAY,
> you
> > can actually keep your connection going. This also allows you to send
> back
> > an HTTP-level error (say 413) which will be easier for people to debug
> when
> > they're faced with a server/intermediary that doesn't want to handle the
> > headers they sent.
>
> But you still need to consume these continuation frames.
>

Just as you need to consume the bytes from the jumbo frame.
The announced size of a frame doesn't change anything w.r.t. the DoS
surface.


>
> > Also, the argument that CONTINUATION support is untested is also
> > unconvincing. I know I've tested my continuation support (both sending
> and
> > receiving), against a few different servers. While I'm sympathetic to the
> > argument that we should remove footguns/untested code, there are always
> > going to be bugs, and this change just trades one footgun (possibly
> > poorly-tested continuation support) for another (almost certainly
> > poorly-tested large frame support).
>
> Yes, but with the difference that it's more in line with common practices
> in various internet protocols. CONTINUATIONS is similar to IP fragments.
> No internet-facing server correctly supports them anymore, they are
> nightmare everywhere, you're torn between dropping them all or trying to
> reassemble them all and suffer from this. So everyone handles them using
> heuristics or hacks. Filling packets with protocol-defined sizes is done
> everywhere (IP, TCP, etc) and tends to be well done.


Given that we're going over an in-order, reliable transport, the analogy
isn't a strong one :)


>
> > > Also concerning the numbers, I too have posted elements (please check
> > > the archives) indicating that video already represents 60% of the
> > > internet traffic and is deemed to 78% by 2018.
> >
> >
> > Let's please not conflate the issue of large headers with video.
>
> The draft is taking about "large frame format" and I thought you were
> responding to that *in general*.
>
> > They're
> > only tangentially related - one is a correctness issue (large headers),
> one
> > is a performance issue. Let's assume some theoretical world where I end
> up
> > supporting large frames (VERY theoretical, at this point). If I don't
> know
> > that I'm receiving video, I don't want to advertise large frame support,
> > because when used poorly, that will kill multiplexing and priority, which
> > are essential to the non-video case. I generally won't know a priori
> > whether a connection I'm making will be used for video or not, so I don't
> > want to offer that performance footgun on every connection I make.
>
> I absolutely agree and that's exactly the purpose here : when you don't
> know for a good reason you want to change the default, it remains
> unchanged.
> However if some of your users start complaining that streaming a video from
> their low-powered home NAS to their TV running your client is slow, you'll
> probably be happy to propose them to experiment with a small change
> allowing
> larger frames between the NAS and the TV so that the video is not jerky
> anymore. At least that's much better than going back here and saying "guys,
> we need to implement an extension, let's cross fingers for it not to break
> intermediaries which are unaware of it, because it will change the
> framing".
>
> Also, telling users that their video is slow because we want to ensure that
> their TV gets a fair share of their home network's bandwidth to their NAS
> will make them laugh at best, given that they're usually watching a single
> video at once on a given screen.


Note that the typical thought is that 'bigger is better', though we know it
to not be true w.r.t. latency.
This is part of the footgun. The good thing about the proposal as written
is that it takes two poor implementations to make for a poor experience,
which makes it likely that users will have adequate experience, at least
for the web case.
The downside of this is that we've added new and interesting failure cases
and latency requirements w.r.t. headers thanks to the buffering
requirements.

I can understand some of the frame-size arguments, though I've obviously
been less worried about them, and worry about adding more complexity to the
base framing (I prefer it be in the part that interprets the frames instead
of the framing itself).
The required-buffering of the whole headers stuff, however, still seems
like a bad idea. It increases the DoS surface area instead of decreasing it.
-=R


>
> > And
> > while video is already a large percentage of the traffic, I haven't seen
> > any numbers so far that convince me it's a large percentage of the
> > connections.
>
> I don't see the relation with connections here. For sure it must be much
> less connections given they're much larger. What matters is end user's
> experience. When a person spends 10 minutes watching a single-connection
> video or 10 minutes reading news and loading 1000 objects, how are some
> connections more important than the other ones ? What is important is
> the number of times the user's experience is badly affected.
>
> > I'll say this again - I think jumbo/large/huge/whatever DATA
> > frames are a perfect candidate for an extension.
>
> And as I explained already, this will never ever happen just because
> you have no way to ensure you can negociate a different framing without
> breaking everything in the middle. TCP window scaling took something
> like 10 or 15 years to take off because intermediaries which started
> to inspect sequence numbers did not know that the window size they were
> looking at was scaled and that it was normal that sequence numbers went
> apparently out of windows. We'll get the exact same issue here with your
> desktop's antivirus, your beloved web site's IPS, etc...
>
> So framing is not a candidate for an extension.
>
> Best regards,
> Willy
>
>
>