Re: HTTP/2 and Websockets

On 25 November 2014 20:28:13 GMT+08:00, Yutaka Hirano <yhirano@google.com> wrote:
>>
>> > 1.1 Each http2 frame payload consists of one-byte WebSocket header
>and
>> >WebSocket data payload.
>> Yes.  Basically there's nowhere to put the ws flags, we discussed and
>they
>> won't fit in the http2 header without dropping some reserved bits,
>which is
>> too restrictive.  So the idea at the moment is stick them as the
>first byte
>> of every ws-mode http2 DATA frame.
>> There's an implied 1.2
>> 1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any
>original
>> ws framing is not respected.  Refragmenting will not be a theoretical
>> corner-case, http2 will make it desirable to cut the ws-carrying DATA
>frame
>> to fit the tx credit on the stream, so it will be normal the frames
>are
>> refragmented to make the best use of the current tx credit for
>minimum
>> latency.  The only heuristic is the last http2 frame-fragment created
>an
>> incoming original ws frame must show the real FIN state of the
>original
>> incoming ws frame, and the others all have FIN=0.
>
>I have no problem with that.
>
>
>> > 2. All intermediaries MUST understand WebSocket (i.e. If there is
>an
>> >intermediary that doesn't understand WebSocket, the opening
>handshake
>> >must fail).
>> Can you live with restricting it to "any intermediary capable of
>rewriting
>> DATA" that does not understand ws-over-http2 must kill the opening
>> handshake?  Because if there's an intermediary like a load balancer,
>we
>> don't really care do we?  It's not going to screw anything up just
>move
>> frames around.
>> How about in http2 spec, ask for
>> a) rewriting intermediaries MUST read HEADERS
>> b) if they see an UPGRADE header they do not know how to deal with,
>they
>> must kill the stream
>> This also covers "upgrade xxx-123" case for http2.
>
>We could ask for it, but I'm not sure if giving a special meaning for
>"rewriting" is suitable from http/2 point of view and it is suitable to
>request at this moment, given that http/2 is almost stabilized (not?).
>cc: Martin.

Thanks.

The problem about making sure the stream upgrade fails if the intermediaries that might corrupt it don't understand the protocol upgraded to isn't ws specific, it's an http2-level issue for any kind of stream upgrades in http2.

Because stream upgrades generally are not forbidden in http2 they're already legit and will place non-http payload in DATA by default.

There's another general problem if the upgrade implies new frame types.  Because without some filtering by intermediaries at upgrade-time, the upgrade handshake will succeed but http2 demands unknown frame types be dropped.  So the end result is a connected but unusable upgrade if any intermediary does not understand the new frame types.  Instead any intermediary who objects to the upgrade should definitively kill it at upgrade handshake time.

Intermediaries that operate at frame level without interpreting or rewriting the payload have no reason to object to a stream upgrade.

Intermediaries that interpret or rewrite the payload

 - must recognize the upgrade header, and

 - either disable their interpretation + rewrite activity on the stream after upgrade to something they don't understand, or disallow the upgrade by killing the stream

Does it sound reasonable?

-Andy

>We had a discussion about the capability check and agreed to use
>SETTINGS
>frames.
>http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/1114.html
>
>>Do you agree with these?
>> I would like to see if we can find a less exclusive solution for the
>> corruption problem.
>> Original ws did not give each intermediary a veto.  It's easy to
>deploy
>> and from my experience reliable to use.  There's a big danger with
>the
>> per-intermediary, default-no, veto, when it's just an adjunct to
>http2 and
>> not part of it, we would define something that will end up
>undeployable.
>
>Originally, I wanted ws-over-http to be on top of http/2 framing layer
>without extending it.
>IIUC that was rejected when END_MESSAGE was removed from the http/2
>spec.
>
>
>On Tue, Nov 25, 2014 at 7:36 PM, Andy Green <andy@warmcat.com> wrote:
>
>>
>>
>> On 25 November 2014 16:16:38 GMT+08:00, Yutaka Hirano
><yhirano@google.com>
>> wrote:
>> >Hmm, OK, I agree that "magic bad proxy" is a magic word.
>>
>> Thanks, I appreciate you gave it some thought.
>>
>> >Let me confirm:
>> > 1 We use http2 DATA frames to convey WebSocket data.
>>
>> Yes.
>>
>> > 1.1 Each http2 frame payload consists of one-byte WebSocket header
>and
>> >WebSocket data payload.
>>
>> Yes.  Basically there's nowhere to put the ws flags, we discussed and
>they
>> won't fit in the http2 header without dropping some reserved bits,
>which is
>> too restrictive.  So the idea at the moment is stick them as the
>first byte
>> of every ws-mode http2 DATA frame.
>>
>> There's an implied 1.2
>>
>> 1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any
>original
>> ws framing is not respected.  Refragmenting will not be a theoretical
>> corner-case, http2 will make it desirable to cut the ws-carrying DATA
>frame
>> to fit the tx credit on the stream, so it will be normal the frames
>are
>> refragmented to make the best use of the current tx credit for
>minimum
>> latency.  The only heuristic is the last http2 frame-fragment created
>an
>> incoming original ws frame must show the real FIN state of the
>original
>> incoming ws frame, and the others all have FIN=0.
>>
>> > 2. All intermediaries MUST understand WebSocket (i.e. If there is
>an
>> >intermediary that doesn't understand WebSocket, the opening
>handshake
>> >must fail).
>>
>> Can you live with restricting it to "any intermediary capable of
>rewriting
>> DATA" that does not understand ws-over-http2 must kill the opening
>> handshake?  Because if there's an intermediary like a load balancer,
>we
>> don't really care do we?  It's not going to screw anything up just
>move
>> frames around.
>>
>> How about in http2 spec, ask for
>>
>> a) rewriting intermediaries MUST read HEADERS
>>
>> b) if they see an UPGRADE header they do not know how to deal with,
>they
>> must kill the stream
>>
>> This also covers "upgrade xxx-123" case for http2.
>>
>> >Do you agree with these?
>>
>> I would like to see if we can find a less exclusive solution for the
>> corruption problem.
>>
>> Original ws did not give each intermediary a veto.  It's easy to
>deploy
>> and from my experience reliable to use.  There's a big danger with
>the
>> per-intermediary, default-no, veto, when it's just an adjunct to
>http2 and
>> not part of it, we would define something that will end up
>undeployable.
>>
>> -Andy
>>
>> >
>> >On Tue, Nov 25, 2014 at 10:42 AM, Andy Green <andy@warmcat.com>
>wrote:
>> >
>> >>
>> >>
>> >> On 25 November 2014 09:18:25 GMT+08:00, Amos Jeffries <
>> >> squid3@treenet.co.nz> wrote:
>> >> >-----BEGIN PGP SIGNED MESSAGE-----
>> >> >Hash: SHA1
>> >> >
>> >> >On 25/11/2014 12:34 p.m., Andy Green wrote:
>> >> >>
>> >> >>
>> >> >> On 22 November 2014 00:54:47 GMT+08:00, Jason Greene
>> >> >> <jason.greene@redhat.com> wrote:
>> >> >>>
>> >> >>>> On Nov 21, 2014, at 5:53 AM, Andy Green <andy@warmcat.com>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On 21 November 2014 18:57:13 GMT+08:00, Amos Jeffries
>> >> >>> <squid3@treenet.co.nz> wrote:
>> >> >>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> >> >>>>>
>> >> >>>>> On 21/11/2014 7:58 p.m., Andy Green wrote:
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> On 21 November 2014 14:41:19 GMT+08:00, Yutaka Hirano
>> >> >>>>>> <yhirano@google.com> wrote:
>> >> >>>>>>> On Fri, Nov 21, 2014 at 12:38 PM, Andy Green
>> >> >>>>>>> <andy@warmcat.com> wrote:
>> >> >>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>> On 21 November 2014 11:30:46 GMT+08:00, Yutaka Hirano
>> >> >>>>>>> <yhirano@google.com>
>> >> >>>>>>>> wrote:
>> >> >>>>>>>>> On Fri, Nov 21, 2014 at 11:42 AM, Andy Green
>> >> >>>>>>>>> <andy@warmcat.com>
>> >> >>>>>>> wrote:
>> >> >>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> On 21 November 2014 10:02:28 GMT+08:00, Yutaka
>> >> >>>>>>>>>> Hirano
>> >> >>>>>>>>> <yhirano@google.com>
>> >> >>>>>>>>>> wrote:
>> >> >>>>>>>>>>>>
>> >> >>>>>>>>>>>> I think you have to take the atomic large ws
>> >> >>>>>>>>>>>> frame thing as a
>> >> >>>>>>>>> genuine
>> >> >>>>>>>>>>>> problem because http2 has the transmit credit
>> >> >>>>>>>>>>>> concept. So even
>> >> >>>>>>> if
>> >> >>>>>>>>>>> you
>> >> >>>>>>>>>>>> buffered one ws frame, you can't sit there
>> >> >>>>>>>>>>>> spewing as much
>> >> >>>>>>> http2
>> >> >>>>>>>>> DATA
>> >> >>>>>>>>>>> frame
>> >> >>>>>>>>>>>> as it needs to atomically encapsulate it,
>> >> >>>>>>>>>>>> without sizing your
>> >> >>>>>>>>> http2
>> >> >>>>>>>>>>> frame
>> >> >>>>>>>>>>>> to suit the tx credit. But it's OK because the
>> >> >>>>>>>>>>>>  implementation can transparently
>> >> >>>>>>> fragment
>> >> >>>>>>>>> the
>> >> >>>>>>>>>>> ws
>> >> >>>>>>>>>>>> data using the ws message semantics... I think
>> >> >>>>>>>>>>>> there's no
>> >> >>>>>>> choice
>> >> >>>>>>>>> but
>> >> >>>>>>>>>>> to
>> >> >>>>>>>>>>>> take that approach.
>> >> >>>>>>>>>>>
>> >> >>>>>>>>>>> Sorry I don't understand what you are proposing.
>> >> >>>>>>>>>>> Can you explain?
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> I'm agreeing with what was already written by
>> >> >>>>>>>>>> someone else on the
>> >> >>>>>>>>> thread.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> Talking about buffering huge ws frames until you
>> >> >>>>>>>>>> have enough to
>> >> >>>>>>> issue
>> >> >>>>>>>>> it
>> >> >>>>>>>>>> all in one big http2 DATA frame will not fly.
>> >> >>>>>>>>>>
>> >> >>>>>>>>>> If you're using this putative ws-over-http2
>> >> >>>>>>>>>> scheme, and you get
>> >> >>>>>>> given
>> >> >>>>>>>>> a
>> >> >>>>>>>>>> huge ws frame to transmit, you should fragment it
>> >> >>>>>>>>>> using RFC6455
>> >> >>>>>>>>> message
>> >> >>>>>>>>>> semantics to some implementation-defined limit
>> >> >>>>>>>>>> that is friendly
>> >> >>>>>>> for
>> >> >>>>>>>>> mux'd
>> >> >>>>>>>>>> http2 transport.
>> >> >>>>>>>>>>
>> >> >>>>>>>>> Thanks.
>> >> >>>>>>>>>
>> >> >>>>>>>>> Strictly speaking, RFC6455 allows an extension to
>> >> >>>>>>>>> give meaning to WebSocket frames, so merging /
>> >> >>>>>>>>> fragmenting frames breaks such extensions. We
>> >> >>>>>>>>> discussed this problem in HyBi and many of us said
>> >> >>>>>>>>> "don't care".
>> >> >>>>>>>>
>> >> >>>>>>>> Yeah extensions except for compression have just not
>> >> >>>>>>>> come into
>> >> >>>>>>> existence.
>> >> >>>>>>>>
>> >> >>>>>>>> So I also see it as don't care.  I'm not even sure
>> >> >>>>>>>> it's true since
>> >> >>>>>>> the
>> >> >>>>>>>> intention during ws discussion was an intermediary can
>> >> >>>>>>>>  fragment
>> >> >>>>>>> frames same
>> >> >>>>>>>> as how tcp packets may be fragmented.
>> >> >>>>>>>>
>> >> >>>>>>>>> In any case, an http/2 frame cannot be bigger than
>> >> >>>>>>>>> 2^24 (or 2^14 without an explicit permission), so I
>> >> >>>>>>>>> think we don't have to worry about DoS.
>> >> >>>>>>>>
>> >> >>>>>>>> ... I don't see that.  If I keep spamming 16MB frames
>> >> >>>>>>>> even on one
>> >> >>>>>>> stream
>> >> >>>>>>>> on a consumer link your latency goes to pieces and if
>> >> >>>>>>>> something is
>> >> >>>>>>> doing
>> >> >>>>>>>> multiple instances of it even a big pipe will feel
>> >> >>>>>>>> pain.
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>> The spec says: Length: The length of the frame payload
>> >> >>>>>>> expressed as an unsigned 24-bit integer. Values greater
>> >> >>>>>>> than 2^14 (16,384) MUST NOT be sent unless the receiver
>> >> >>>>>>> has set a larger value for SETTINGS_MAX_FRAME_SIZE. So
>> >> >>>>>>> by default 2^14 is the upper limit. Larger frames will be
>> >> >>>>>>> used only if the receiver wants.
>> >> >>>>>>
>> >> >>>>>> Okay.  But if buffering whole ws frames no matter how big
>> >> >>>>>> was the plan, and the one end wanted to send a 16MB ws
>> >> >>>>>> frame, the other end would say, "sure" you can send me >16K
>> >> >>>>>> and he'd spam his 16MB frames.  Default 16K limit doesn't
>> >> >>>>>> help much because the server is complicit in thinking
>> >> >>>>>> sending huge ws frames is good.
>> >> >>>>>>
>> >> >>>>>
>> >> >>>>> You switched between "receiver" and "server" there in your
>> >> >>>>> concepts.
>> >> >>>>
>> >> >>>> Well okay, it's not my day job.
>> >> >>>>
>> >> >>>>> Overlooking the fact that intermediaries in HTTP can cut
>> >> >>>>> DATA up
>> >> >>> into
>> >> >>>>> smaller frames and stitch them together into larger ones as
>> >> >>>>> needed hop-by-hop. So what your sender gets told it can send
>> >> >>>>> on the client->proxy connection may not be what gets
>> >> >>>>> delivered to the
>> >> >>> server.
>> >> >>>>>
>> >> >>>>> That was the major reason for WSDATA not being HTTP/2 DATA.
>> >> >>>>> That
>> >> >>> flags
>> >> >>>>> byte being mentioned at the beginning of DATA payload will
>> >> >>>>> get
>> >> >>> screwed
>> >> >>>>> over and moved around too much. Unless it is at fixed
>> >> >>>>> positions of
>> >> >>> the
>> >> >>>>> total WS stream inside the DATA payloads.
>> >> >>>>
>> >> >>>> What does 'too much' mean... it should not attempt to
>preserve
>> >> >>>> the
>> >> >>> original ws frame sizes, just send http2 frames of a size that
>> >> >>> suit it when an amount of ws payload data that suits it has
>> >> >>> appeared. So the bit of tx logic that chops up the ws payload
>> >> >>> data, takes care of deferring FIN until the last chunk.  And
>the
>> >> >>> receiver doesn't have to do anything just pass up the frames
>it
>> >> >>> got with the ws flags it got.
>> >> >>>>
>> >> >>>> What's the problem?
>> >> >>>>
>> >> >>>>> But then, as has been mentioned since HEADERS initiated a WS
>> >> >>>>> stream, why is DATA needing to be re-flagged specially? it
>> >> >>>>> is opaque payload bytes of the stream already flagged as
>> >> >>>>> being a WS transaction.
>> >> >>>>
>> >> >>>> I think it's alright being an implicit attribute of the
>> >> >>>> stream.
>> >> >>>>
>> >> >>>> But I understood the general objection from Hirano-san, it's
>> >> >>>> that
>> >> >>> there might be dumb proxies who didn't follow the update
>> >> >>> activity on that stream and might parse what they see there
>as...
>> >> >>> http2 data.
>> >> >>>>
>> >> >>>> 'Magic bad proxies' was a stick that extinguished a lot of
>> >> >>>> discussion
>> >> >>> on hybi.... he's not going to get fired for shaking that stick
>> >> >>> at us. It doesn't damage the argument to point out there's no
>> >> >>> evidence the bad proxy exists.  But if it's WSDATA and http2
>> >> >>> wraps up without containing WSDATA this stuff will never get
>> >> >>> deployed, because if anybody doesn't understand WSDATA the
>> >> >>> connection won't work even if the handshake does, lol
>> >> >>>
>> >> >>> A bad proxy that corrupts DATA frames is going to break more
>> >> >>> than web sockets, it will also break common HTTP use cases.
>> >> >>
>> >> >> Yeah.
>> >> >>
>> >> >> So what's the plan to get websockets working on HTTP2?  This is
>> >> >> the place where we're supposed to be sorting that out.  But
>it's
>> >> >> looking stalled.
>> >> >>
>> >> >> Since there is no active plan right now, the default situation
>is
>> >> >> a big nothing and perhaps a big crappy rollout of WSDATA frames
>> >> >> that nobody supports leading to negotiated and then dead WS
>> >> >> connections.
>> >> >>
>> >> >> Is that an OK result for everybody?
>> >> >>
>> >> >> As discussed - nobody disagreed with the logical workability of
>> >> >> this - it can be done concisely over DATA today with zero
>change
>> >or
>> >> >> delay to HTTP2.
>> >> >>
>> >> >> The objection from Hirano-san is if we put [1 byte WS flags] +
>[
>> >> >> WS payload ] in HTTP2 DATA frames, "a bad proxy / intermediary"
>> >> >> may think it's OK to mangle the content of DATA frames as if it
>> >was
>> >> >> HTTP content.
>> >> >>
>> >> >> Personally I don't think it's realistic, but I saw this
>argument
>> >> >> block and destroy things on hybi, so we have to shrug and
>pretend
>> >> >> it is a serious issue.  One of the reasons this argument is so
>> >> >> damaging and unhelpful is when you ask, "how broken is that
>proxy
>> >> >> exactly", there is never any limit to how deranged it is in
>> >theory,
>> >> >> because it can be as deranged as you like, and the guy
>proposing
>> >it
>> >> >> does not want to give up any ground by saying, "well, I guess
>it
>> >> >> might not flip bits randomly then", or "yes it must understand
>> >http
>> >> >> headers". So we get into arguments about the characteristics of
>> >the
>> >> >> nonexistant magic broken proxy rather than do engineering.
>> >> >
>> >> >I think Hirano-san's proxy case is a good one to work with. But
>> >limit
>> >> >it to being a HTTP/2 compliant proxy. That limits it to
>> >> >slicing/aggregating DATA frames any way it likes but not
>re-ordering
>> >> >or changing payload bytes unless there are transformations which
>the
>> >> >HTTP/2 headers permit (the TCP analogy).
>> >>
>> >> Well I agree, you're basically saying it has to survive working
>over
>> >an
>> >> http2 proxy, that's clearly required.
>> >>
>> >> Hinaro-san is saying something further than that, which is, to his
>> >mind,
>> >> every proxy must be able to look inside every DATA packet and
>> >understand it
>> >> according to an HTTP state (post-upgrade to ws) that it may be
>> >incapable of
>> >> understanding.
>> >>
>> >> That magic broken proxy is in trouble anyway because nothing stops
>me
>> >> opening an http2 stream and negotiate an upgrade on it to xxx-123
>> >protocol
>> >> and then all the http2 DATA packets are full of xxx-123 protocol
>> >data,
>> >> without needing any blessing from anyone.
>> >>
>> >> It's already legal in http2 so his objection seems to shine no
>light
>> >on
>> >> anything.
>> >>
>> >> >A truely broken magic proxy does not exist yet in any HTTP/2
>> >> >implementations and HTTP/2 itself is taking the "hard line" of
>> >> >defining RST_STREAM or GOAWAY to be used when any kind of binary
>> >level
>> >> >broken proxy is detected.
>> >> >
>> >> >So the magic proxy can take your 1-byte flags + 6 byte WS payload
>> >DATA
>> >> >frame and slice it. Delivering 1 byte DATA frame followed by 6
>byte
>> >> >DATA frame to the server. This is particularly likely to happen
>if
>> >> >your WS client writes the flags and WS payload in separate
>> >operations
>> >> >to the HTTP/2 library/backend.
>> >>
>> >> Yes... the parsing code must be immune to fragmentation attacks
>> >anyway,
>> >> this is no extra burden than that.
>> >>
>> >> > - all you need to resolve this is a length+flags+payload
>structure
>> >to
>> >> >the WS data. You can do that at the HTTP//2 stream level, or
>inside
>> >> >the DATA frame payload.
>> >> > - if you dont know the length in advance then stream level is
>best
>> >> >for HTTP/2 compatibility.
>> >>
>> >> I didn't really understand what you meant here.
>> >>
>> >> >From what was discussed, the frame length is always known since
>it's
>> >in
>> >> the http2 frame.  The message length can be indeterminite (or
>> >infinite) in
>> >> ws.
>> >>
>> >> -Andy
>> >>
>> >> >Amos
>> >> >-----BEGIN PGP SIGNATURE-----
>> >> >Version: GnuPG v2.0.22 (MingW32)
>> >> >
>> >> >iQEcBAEBAgAGBQJUc9jhAAoJELJo5wb/XPRjDsAH/21qGwDJVC2iOEYDeBEb5rBG
>> >> >kgYAeMri9yLrRis/4Pv0QtEQgxxMNVkWxDP+NNKmwCkhz3dF3X/y0v8ryXyr40aE
>> >> >Kw6+m5vsLEtQ1/d7Xxbkz8iqJBZsM/1fklRGJ2G0YKLUED4KDf2hKmnkQwbDi7la
>> >> >LUV1t6ij50gT9VmfW6r0dUwnLOt2M9qYRbTbGmgQbuc/6Zsqqp1VNtSOTpG/HC3+
>> >> >xl3iZCDTJYyi01lNXZNsqzlXrx8zLT0i7KdE+QM+Vg/v708rUAp8sF26jxzsB/M2
>> >> >d9oL2jXBJQ4ydh16Od5lwJrKlCqK7QHg3QHxRe8wmP9dabzHUVef3QmqPtoXJwg=
>> >> >=9I96
>> >> >-----END PGP SIGNATURE-----
>> >>
>> >>
>> >>
>>
>>

Received on Tuesday, 25 November 2014 13:23:51 UTC