Re: HTTP/2: allow binary data in header field values

Hi Amos,

On Mon, Aug 28, 2017 at 10:28 PM, Amos Jeffries <squid3@treenet.co.nz> wrote:
> The prohibition of these characters is a Security Considerations requirement
> in HTTP. It would be best to keep that fact clearly up-front. It was not a
> casual / arbitrary design decision, so the reasons for it cannot just be
> ignored when implementing or negotiating extended behaviour.
>
> So long as HTTP/1 <-> HTTP/2 gateways exist the security attacks will remain
> a problem. This is not a theoretical problem either, intermediaries are
> still fending off active attacks and malformed agent messages involving
> these three characters in HTTP/1.x environment before HTTP/2 mapping even
> gets involved.
>
> The simple problem is that one cannot guarantee the absence of a mapping
> gateway in any transaction. So it HAS to be considered by every agent
> involved.

100% agreed. What I meant is that it's an artificial limitation in the
HTTP/2 protocol, since it wouldn't be necessary in HTTP/2 only world.

> It is worth noting that base64 encoding is more efficiently encoded by
> HPACK. So avoiding it is a con', not a pro'.

I'm not sure that I agree.

Base64 with Huffman Encoding is basically a wash in terms of space
(4/3 * 6 bits ~= 1 byte per byte), and requires somehow CPU-intensive
transformations.

Binary data without Huffman Encoding is also 1 byte per byte, and
requires no CPU-intensive transformations.

> There is no need for this. With the SETTINGS value already negotiating the
> ability HPACK simply needs to decode the wire syntax into a binary 'string'
> header.
>
> Agents that comply and reject the headers will not be negotiating to accept
> them. If the binary value sent in any particular message header does not use
> these trouble characters there is no harm in letting it through, so
> artificially forcing rejection is not beneficial here like the RFC 7540
> requirement was for default / general use.

Oh, it wasn't meant to force rejection. The main point of NUL byte
(0x00) is for binary-aware proxies to know which headers they need to
base64 encode when forwarding to HTTP/1.x or HTTP/2 peers unaware of
this extension.

> As written that would violate RFC 7540. This requirement needs to take the
> form of prohibiting sending binary headers to any peers which has not
> explicitly negotiated the extension being defined.
>  ie, comply with RFC7540 an all connections unless explicitly negotiated
> otherwise on a per-connection basis.

Yes, this wasn't formal enough language, but I meant the same.

> Of course they can. Every coding language has some form of array or
> linked-list structure available.
>
> To display these type of header in *ASCII MiME format* on the other hand
> requires encoding by the display agent. HTTP/2 does not change any
> requirements around display, it is concerned only with the on-wire delivery.

Fair enough. We could invent a format to allow concatenation of those
header field values, but I'm not sure if that's worth the trouble.

> How do you plan on making it "native HTTP/2" without replacing the whole
> HTTP/2 RFC *and* getting that new specification rolled out to the non-QUIC
> world?
>
> (I am Seriously interested in that answer. Many of us middleware
> implementers have been pushing for UTF-8 / binary support in headers for
> around 10 years already and progress has been painfully slow).
>
> It seems to me you [like several of us] are dreaming of HTTP/3-over-QUIC.
> Not HTTP/2-over-QUIC, extended or otherwise. I am very doubtful that getting
> all this done before QUIC rolls out is going to be possible - a negotiable
> extension is far more realistic and will allow a testing rollout to happen
> before everybody in the HTTP world has to change code for it.

I'm not sure if that was clear from my original email (it probably
wasn't...), but the proposal is to allow binary header field values
(prefixed by NUL byte) along regular VCHAR headers, not to convert
everything (dates, sizes, etc.) into binary values.

As for "native" - what I meant is that in HTTP/2-over-TCP, allowing
binary header field values will be announced via HTTP/2 SETTINGS
option, and, if we're luckily enough and this gets standardized before
QUIC, then in HTTP/2-over-QUIC, this settings would be supported
natively (i.e. simply enabled by default), so that all
HTTP/2-over-QUIC peers would be binary-aware.

I don't believe this extension requires a lot of changes (assuming
that w're not converting everything to binary), and HTTP/2-over-QUIC
is already different enough that this should be an acceptable change.

Best regards,
Piotr Sikora

Received on Tuesday, 29 August 2017 07:27:15 UTC