Re: HTTP/2 and Websockets

>
> > 1.1 Each http2 frame payload consists of one-byte WebSocket header and
> >WebSocket data payload.
> Yes.  Basically there's nowhere to put the ws flags, we discussed and they
> won't fit in the http2 header without dropping some reserved bits, which is
> too restrictive.  So the idea at the moment is stick them as the first byte
> of every ws-mode http2 DATA frame.
> There's an implied 1.2
> 1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any original
> ws framing is not respected.  Refragmenting will not be a theoretical
> corner-case, http2 will make it desirable to cut the ws-carrying DATA frame
> to fit the tx credit on the stream, so it will be normal the frames are
> refragmented to make the best use of the current tx credit for minimum
> latency.  The only heuristic is the last http2 frame-fragment created an
> incoming original ws frame must show the real FIN state of the original
> incoming ws frame, and the others all have FIN=0.

I have no problem with that.


> > 2. All intermediaries MUST understand WebSocket (i.e. If there is an
> >intermediary that doesn't understand WebSocket, the opening handshake
> >must fail).
> Can you live with restricting it to "any intermediary capable of rewriting
> DATA" that does not understand ws-over-http2 must kill the opening
> handshake?  Because if there's an intermediary like a load balancer, we
> don't really care do we?  It's not going to screw anything up just move
> frames around.
> How about in http2 spec, ask for
> a) rewriting intermediaries MUST read HEADERS
> b) if they see an UPGRADE header they do not know how to deal with, they
> must kill the stream
> This also covers "upgrade xxx-123" case for http2.

We could ask for it, but I'm not sure if giving a special meaning for
"rewriting" is suitable from http/2 point of view and it is suitable to
request at this moment, given that http/2 is almost stabilized (not?).
cc: Martin.
We had a discussion about the capability check and agreed to use SETTINGS
frames.
http://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/1114.html

>Do you agree with these?
> I would like to see if we can find a less exclusive solution for the
> corruption problem.
> Original ws did not give each intermediary a veto.  It's easy to deploy
> and from my experience reliable to use.  There's a big danger with the
> per-intermediary, default-no, veto, when it's just an adjunct to http2 and
> not part of it, we would define something that will end up undeployable.

Originally, I wanted ws-over-http to be on top of http/2 framing layer
without extending it.
IIUC that was rejected when END_MESSAGE was removed from the http/2 spec.


On Tue, Nov 25, 2014 at 7:36 PM, Andy Green <andy@warmcat.com> wrote:

>
>
> On 25 November 2014 16:16:38 GMT+08:00, Yutaka Hirano <yhirano@google.com>
> wrote:
> >Hmm, OK, I agree that "magic bad proxy" is a magic word.
>
> Thanks, I appreciate you gave it some thought.
>
> >Let me confirm:
> > 1 We use http2 DATA frames to convey WebSocket data.
>
> Yes.
>
> > 1.1 Each http2 frame payload consists of one-byte WebSocket header and
> >WebSocket data payload.
>
> Yes.  Basically there's nowhere to put the ws flags, we discussed and they
> won't fit in the http2 header without dropping some reserved bits, which is
> too restrictive.  So the idea at the moment is stick them as the first byte
> of every ws-mode http2 DATA frame.
>
> There's an implied 1.2
>
> 1.2  Every ws-mode http2 DATA is always a complete ws frame.  Any original
> ws framing is not respected.  Refragmenting will not be a theoretical
> corner-case, http2 will make it desirable to cut the ws-carrying DATA frame
> to fit the tx credit on the stream, so it will be normal the frames are
> refragmented to make the best use of the current tx credit for minimum
> latency.  The only heuristic is the last http2 frame-fragment created an
> incoming original ws frame must show the real FIN state of the original
> incoming ws frame, and the others all have FIN=0.
>
> > 2. All intermediaries MUST understand WebSocket (i.e. If there is an
> >intermediary that doesn't understand WebSocket, the opening handshake
> >must fail).
>
> Can you live with restricting it to "any intermediary capable of rewriting
> DATA" that does not understand ws-over-http2 must kill the opening
> handshake?  Because if there's an intermediary like a load balancer, we
> don't really care do we?  It's not going to screw anything up just move
> frames around.
>
> How about in http2 spec, ask for
>
> a) rewriting intermediaries MUST read HEADERS
>
> b) if they see an UPGRADE header they do not know how to deal with, they
> must kill the stream
>
> This also covers "upgrade xxx-123" case for http2.
>
> >Do you agree with these?
>
> I would like to see if we can find a less exclusive solution for the
> corruption problem.
>
> Original ws did not give each intermediary a veto.  It's easy to deploy
> and from my experience reliable to use.  There's a big danger with the
> per-intermediary, default-no, veto, when it's just an adjunct to http2 and
> not part of it, we would define something that will end up undeployable.
>
> -Andy
>
> >
> >On Tue, Nov 25, 2014 at 10:42 AM, Andy Green <andy@warmcat.com> wrote:
> >
> >>
> >>
> >> On 25 November 2014 09:18:25 GMT+08:00, Amos Jeffries <
> >> squid3@treenet.co.nz> wrote:
> >> >-----BEGIN PGP SIGNED MESSAGE-----
> >> >Hash: SHA1
> >> >
> >> >On 25/11/2014 12:34 p.m., Andy Green wrote:
> >> >>
> >> >>
> >> >> On 22 November 2014 00:54:47 GMT+08:00, Jason Greene
> >> >> <jason.greene@redhat.com> wrote:
> >> >>>
> >> >>>> On Nov 21, 2014, at 5:53 AM, Andy Green <andy@warmcat.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On 21 November 2014 18:57:13 GMT+08:00, Amos Jeffries
> >> >>> <squid3@treenet.co.nz> wrote:
> >> >>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
> >> >>>>>
> >> >>>>> On 21/11/2014 7:58 p.m., Andy Green wrote:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On 21 November 2014 14:41:19 GMT+08:00, Yutaka Hirano
> >> >>>>>> <yhirano@google.com> wrote:
> >> >>>>>>> On Fri, Nov 21, 2014 at 12:38 PM, Andy Green
> >> >>>>>>> <andy@warmcat.com> wrote:
> >> >>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On 21 November 2014 11:30:46 GMT+08:00, Yutaka Hirano
> >> >>>>>>> <yhirano@google.com>
> >> >>>>>>>> wrote:
> >> >>>>>>>>> On Fri, Nov 21, 2014 at 11:42 AM, Andy Green
> >> >>>>>>>>> <andy@warmcat.com>
> >> >>>>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> On 21 November 2014 10:02:28 GMT+08:00, Yutaka
> >> >>>>>>>>>> Hirano
> >> >>>>>>>>> <yhirano@google.com>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I think you have to take the atomic large ws
> >> >>>>>>>>>>>> frame thing as a
> >> >>>>>>>>> genuine
> >> >>>>>>>>>>>> problem because http2 has the transmit credit
> >> >>>>>>>>>>>> concept. So even
> >> >>>>>>> if
> >> >>>>>>>>>>> you
> >> >>>>>>>>>>>> buffered one ws frame, you can't sit there
> >> >>>>>>>>>>>> spewing as much
> >> >>>>>>> http2
> >> >>>>>>>>> DATA
> >> >>>>>>>>>>> frame
> >> >>>>>>>>>>>> as it needs to atomically encapsulate it,
> >> >>>>>>>>>>>> without sizing your
> >> >>>>>>>>> http2
> >> >>>>>>>>>>> frame
> >> >>>>>>>>>>>> to suit the tx credit. But it's OK because the
> >> >>>>>>>>>>>>  implementation can transparently
> >> >>>>>>> fragment
> >> >>>>>>>>> the
> >> >>>>>>>>>>> ws
> >> >>>>>>>>>>>> data using the ws message semantics... I think
> >> >>>>>>>>>>>> there's no
> >> >>>>>>> choice
> >> >>>>>>>>> but
> >> >>>>>>>>>>> to
> >> >>>>>>>>>>>> take that approach.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Sorry I don't understand what you are proposing.
> >> >>>>>>>>>>> Can you explain?
> >> >>>>>>>>>>
> >> >>>>>>>>>> I'm agreeing with what was already written by
> >> >>>>>>>>>> someone else on the
> >> >>>>>>>>> thread.
> >> >>>>>>>>>>
> >> >>>>>>>>>> Talking about buffering huge ws frames until you
> >> >>>>>>>>>> have enough to
> >> >>>>>>> issue
> >> >>>>>>>>> it
> >> >>>>>>>>>> all in one big http2 DATA frame will not fly.
> >> >>>>>>>>>>
> >> >>>>>>>>>> If you're using this putative ws-over-http2
> >> >>>>>>>>>> scheme, and you get
> >> >>>>>>> given
> >> >>>>>>>>> a
> >> >>>>>>>>>> huge ws frame to transmit, you should fragment it
> >> >>>>>>>>>> using RFC6455
> >> >>>>>>>>> message
> >> >>>>>>>>>> semantics to some implementation-defined limit
> >> >>>>>>>>>> that is friendly
> >> >>>>>>> for
> >> >>>>>>>>> mux'd
> >> >>>>>>>>>> http2 transport.
> >> >>>>>>>>>>
> >> >>>>>>>>> Thanks.
> >> >>>>>>>>>
> >> >>>>>>>>> Strictly speaking, RFC6455 allows an extension to
> >> >>>>>>>>> give meaning to WebSocket frames, so merging /
> >> >>>>>>>>> fragmenting frames breaks such extensions. We
> >> >>>>>>>>> discussed this problem in HyBi and many of us said
> >> >>>>>>>>> "don't care".
> >> >>>>>>>>
> >> >>>>>>>> Yeah extensions except for compression have just not
> >> >>>>>>>> come into
> >> >>>>>>> existence.
> >> >>>>>>>>
> >> >>>>>>>> So I also see it as don't care.  I'm not even sure
> >> >>>>>>>> it's true since
> >> >>>>>>> the
> >> >>>>>>>> intention during ws discussion was an intermediary can
> >> >>>>>>>>  fragment
> >> >>>>>>> frames same
> >> >>>>>>>> as how tcp packets may be fragmented.
> >> >>>>>>>>
> >> >>>>>>>>> In any case, an http/2 frame cannot be bigger than
> >> >>>>>>>>> 2^24 (or 2^14 without an explicit permission), so I
> >> >>>>>>>>> think we don't have to worry about DoS.
> >> >>>>>>>>
> >> >>>>>>>> ... I don't see that.  If I keep spamming 16MB frames
> >> >>>>>>>> even on one
> >> >>>>>>> stream
> >> >>>>>>>> on a consumer link your latency goes to pieces and if
> >> >>>>>>>> something is
> >> >>>>>>> doing
> >> >>>>>>>> multiple instances of it even a big pipe will feel
> >> >>>>>>>> pain.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>> The spec says: Length: The length of the frame payload
> >> >>>>>>> expressed as an unsigned 24-bit integer. Values greater
> >> >>>>>>> than 2^14 (16,384) MUST NOT be sent unless the receiver
> >> >>>>>>> has set a larger value for SETTINGS_MAX_FRAME_SIZE. So
> >> >>>>>>> by default 2^14 is the upper limit. Larger frames will be
> >> >>>>>>> used only if the receiver wants.
> >> >>>>>>
> >> >>>>>> Okay.  But if buffering whole ws frames no matter how big
> >> >>>>>> was the plan, and the one end wanted to send a 16MB ws
> >> >>>>>> frame, the other end would say, "sure" you can send me >16K
> >> >>>>>> and he'd spam his 16MB frames.  Default 16K limit doesn't
> >> >>>>>> help much because the server is complicit in thinking
> >> >>>>>> sending huge ws frames is good.
> >> >>>>>>
> >> >>>>>
> >> >>>>> You switched between "receiver" and "server" there in your
> >> >>>>> concepts.
> >> >>>>
> >> >>>> Well okay, it's not my day job.
> >> >>>>
> >> >>>>> Overlooking the fact that intermediaries in HTTP can cut
> >> >>>>> DATA up
> >> >>> into
> >> >>>>> smaller frames and stitch them together into larger ones as
> >> >>>>> needed hop-by-hop. So what your sender gets told it can send
> >> >>>>> on the client->proxy connection may not be what gets
> >> >>>>> delivered to the
> >> >>> server.
> >> >>>>>
> >> >>>>> That was the major reason for WSDATA not being HTTP/2 DATA.
> >> >>>>> That
> >> >>> flags
> >> >>>>> byte being mentioned at the beginning of DATA payload will
> >> >>>>> get
> >> >>> screwed
> >> >>>>> over and moved around too much. Unless it is at fixed
> >> >>>>> positions of
> >> >>> the
> >> >>>>> total WS stream inside the DATA payloads.
> >> >>>>
> >> >>>> What does 'too much' mean... it should not attempt to preserve
> >> >>>> the
> >> >>> original ws frame sizes, just send http2 frames of a size that
> >> >>> suit it when an amount of ws payload data that suits it has
> >> >>> appeared. So the bit of tx logic that chops up the ws payload
> >> >>> data, takes care of deferring FIN until the last chunk.  And the
> >> >>> receiver doesn't have to do anything just pass up the frames it
> >> >>> got with the ws flags it got.
> >> >>>>
> >> >>>> What's the problem?
> >> >>>>
> >> >>>>> But then, as has been mentioned since HEADERS initiated a WS
> >> >>>>> stream, why is DATA needing to be re-flagged specially? it
> >> >>>>> is opaque payload bytes of the stream already flagged as
> >> >>>>> being a WS transaction.
> >> >>>>
> >> >>>> I think it's alright being an implicit attribute of the
> >> >>>> stream.
> >> >>>>
> >> >>>> But I understood the general objection from Hirano-san, it's
> >> >>>> that
> >> >>> there might be dumb proxies who didn't follow the update
> >> >>> activity on that stream and might parse what they see there as...
> >> >>> http2 data.
> >> >>>>
> >> >>>> 'Magic bad proxies' was a stick that extinguished a lot of
> >> >>>> discussion
> >> >>> on hybi.... he's not going to get fired for shaking that stick
> >> >>> at us. It doesn't damage the argument to point out there's no
> >> >>> evidence the bad proxy exists.  But if it's WSDATA and http2
> >> >>> wraps up without containing WSDATA this stuff will never get
> >> >>> deployed, because if anybody doesn't understand WSDATA the
> >> >>> connection won't work even if the handshake does, lol
> >> >>>
> >> >>> A bad proxy that corrupts DATA frames is going to break more
> >> >>> than web sockets, it will also break common HTTP use cases.
> >> >>
> >> >> Yeah.
> >> >>
> >> >> So what's the plan to get websockets working on HTTP2?  This is
> >> >> the place where we're supposed to be sorting that out.  But it's
> >> >> looking stalled.
> >> >>
> >> >> Since there is no active plan right now, the default situation is
> >> >> a big nothing and perhaps a big crappy rollout of WSDATA frames
> >> >> that nobody supports leading to negotiated and then dead WS
> >> >> connections.
> >> >>
> >> >> Is that an OK result for everybody?
> >> >>
> >> >> As discussed - nobody disagreed with the logical workability of
> >> >> this - it can be done concisely over DATA today with zero change
> >or
> >> >> delay to HTTP2.
> >> >>
> >> >> The objection from Hirano-san is if we put [1 byte WS flags] + [
> >> >> WS payload ] in HTTP2 DATA frames, "a bad proxy / intermediary"
> >> >> may think it's OK to mangle the content of DATA frames as if it
> >was
> >> >> HTTP content.
> >> >>
> >> >> Personally I don't think it's realistic, but I saw this argument
> >> >> block and destroy things on hybi, so we have to shrug and pretend
> >> >> it is a serious issue.  One of the reasons this argument is so
> >> >> damaging and unhelpful is when you ask, "how broken is that proxy
> >> >> exactly", there is never any limit to how deranged it is in
> >theory,
> >> >> because it can be as deranged as you like, and the guy proposing
> >it
> >> >> does not want to give up any ground by saying, "well, I guess it
> >> >> might not flip bits randomly then", or "yes it must understand
> >http
> >> >> headers". So we get into arguments about the characteristics of
> >the
> >> >> nonexistant magic broken proxy rather than do engineering.
> >> >
> >> >I think Hirano-san's proxy case is a good one to work with. But
> >limit
> >> >it to being a HTTP/2 compliant proxy. That limits it to
> >> >slicing/aggregating DATA frames any way it likes but not re-ordering
> >> >or changing payload bytes unless there are transformations which the
> >> >HTTP/2 headers permit (the TCP analogy).
> >>
> >> Well I agree, you're basically saying it has to survive working over
> >an
> >> http2 proxy, that's clearly required.
> >>
> >> Hinaro-san is saying something further than that, which is, to his
> >mind,
> >> every proxy must be able to look inside every DATA packet and
> >understand it
> >> according to an HTTP state (post-upgrade to ws) that it may be
> >incapable of
> >> understanding.
> >>
> >> That magic broken proxy is in trouble anyway because nothing stops me
> >> opening an http2 stream and negotiate an upgrade on it to xxx-123
> >protocol
> >> and then all the http2 DATA packets are full of xxx-123 protocol
> >data,
> >> without needing any blessing from anyone.
> >>
> >> It's already legal in http2 so his objection seems to shine no light
> >on
> >> anything.
> >>
> >> >A truely broken magic proxy does not exist yet in any HTTP/2
> >> >implementations and HTTP/2 itself is taking the "hard line" of
> >> >defining RST_STREAM or GOAWAY to be used when any kind of binary
> >level
> >> >broken proxy is detected.
> >> >
> >> >So the magic proxy can take your 1-byte flags + 6 byte WS payload
> >DATA
> >> >frame and slice it. Delivering 1 byte DATA frame followed by 6 byte
> >> >DATA frame to the server. This is particularly likely to happen if
> >> >your WS client writes the flags and WS payload in separate
> >operations
> >> >to the HTTP/2 library/backend.
> >>
> >> Yes... the parsing code must be immune to fragmentation attacks
> >anyway,
> >> this is no extra burden than that.
> >>
> >> > - all you need to resolve this is a length+flags+payload structure
> >to
> >> >the WS data. You can do that at the HTTP//2 stream level, or inside
> >> >the DATA frame payload.
> >> > - if you dont know the length in advance then stream level is best
> >> >for HTTP/2 compatibility.
> >>
> >> I didn't really understand what you meant here.
> >>
> >> >From what was discussed, the frame length is always known since it's
> >in
> >> the http2 frame.  The message length can be indeterminite (or
> >infinite) in
> >> ws.
> >>
> >> -Andy
> >>
> >> >Amos
> >> >-----BEGIN PGP SIGNATURE-----
> >> >Version: GnuPG v2.0.22 (MingW32)
> >> >
> >> >iQEcBAEBAgAGBQJUc9jhAAoJELJo5wb/XPRjDsAH/21qGwDJVC2iOEYDeBEb5rBG
> >> >kgYAeMri9yLrRis/4Pv0QtEQgxxMNVkWxDP+NNKmwCkhz3dF3X/y0v8ryXyr40aE
> >> >Kw6+m5vsLEtQ1/d7Xxbkz8iqJBZsM/1fklRGJ2G0YKLUED4KDf2hKmnkQwbDi7la
> >> >LUV1t6ij50gT9VmfW6r0dUwnLOt2M9qYRbTbGmgQbuc/6Zsqqp1VNtSOTpG/HC3+
> >> >xl3iZCDTJYyi01lNXZNsqzlXrx8zLT0i7KdE+QM+Vg/v708rUAp8sF26jxzsB/M2
> >> >d9oL2jXBJQ4ydh16Od5lwJrKlCqK7QHg3QHxRe8wmP9dabzHUVef3QmqPtoXJwg=
> >> >=9I96
> >> >-----END PGP SIGNATURE-----
> >>
> >>
> >>
>
>