W3C home > Mailing lists > Public > ietf-http-wg@w3.org > July to September 2021

Negotiating Window Limits for Content Encodings

From: W. Felix Handte <w@felixhandte.com>
Date: Wed, 4 Aug 2021 17:47:00 -0400
To: HTTP Working Group <ietf-http-wg@w3.org>
Cc: Yann Collet <cyan@fb.com>, Nick Terrell <terrelln@fb.com>, klauspost@gmail.com
Message-ID: <f4e0a0f4-f74b-7a08-d704-6c70930cc06d@felixhandte.com>
Hello all,

I hope you had a pleasant IETF 111.

TL;DR:

I want to specify an option along these lines:

  Accept-Encoding: zstd;w=18

Which in this case would mean that a client will accept a
zstd-compressed response, only if the window size is less than or equal
to 256KB (2 ** 18 bytes).

Background:

Back in the old days, decompression had well-known memory
requirements--the DEFLATE family of compressors are limited to a 32KB
window, so you were guaranteed that that (plus some other small
datastructures) was all the memory you'd need to maintain the
decompressor state for a response, no matter what came down the pipe.

One of the ways modern compressors improve on DEFLATE is by supporting
much larger windows. Brotli supports windows up to 16MB [0], though the
non-standard mode "Large Window Brotli" extends that to 1GB. Zstandard
theoretically supports 3.75TB windows [1], though in practice they're
limited to 2GB [2].

In the absence of any coordination mechanism, when a client indicates
they accept these encodings, a server is allowed to send back a payload
that would require a (very) large amount of memory to decode, which
might lead to resource exhaustion or a failure to decode the payload.
(E.g., the zstd library by default will reject a frame that requires
more than 128MB to decode, unless you explicitly allow larger windows [3].)

Currently the best resolution is mutual caution. The Zstandard RFC
provides interoperability guidance that suggests that recipients support
at least an 8 MB window and that senders avoid using a window larger
than 8 MB [1]. To my knowledge, Brotli doesn't provide any guidance.

It has become apparent that this is insufficient to the variety of
environments that HTTP is used in. For the transfer of very large
resources (esp. e.g. patch files), large windows can be crucial to good
compression performance. So we should allow very large windows when
useful. But for mobile or embedded or highly parallel applications,
we've seen that a compromise value like 8 MB is already too large, and
they may want to express limits <1MB.

Proposal:

I want to give endpoints an explicit coordination mechanism for
selecting compression window sizes. I think the most straightforward way
to do that is to add an option to the content-coding identifiers
advertised in an Accept-Encoding header, so that recipients can
advertise the largest window they are willing to handle for each
encoding. Senders would then be asked to either restrict themselves to
generating a response with a window no larger than the limit or just not
use that encoding.

Specifically, I'm thinking of that being the "w" option, which has a
positive integer value which is the log_2 of the maximum window the
client will accept for that encoding. (See the example at the beginning.)

I think that what I am proposing would be meaningfully better than the
current state of affairs. At the same time, I have several questions /
uncertainties for which your advice would be most welcome.

So before I put this in I-D language, I wanted to circulate the idea here.

Some of those Questions:

1. Is this worth making a generic mechanism and specifying it as an
option for any content-coding for which it's meaningful? Or should I
just restrict this to Zstd, which is where I'm aware of the most need?

2. Is this the best place to insert such a mechanism? Or should it be
its own header? (Would that be easier for caches to handle?)

3. Is it effective to tack on an option like this to existing
content-coding identifiers? Since the existing installed base does not
produce or recognize this option, that means it can only ever be best
effort for existing content-codings. A heavy-weight alternative would be
to introduce new identifiers that mandate handling this signal, but I
don't want to go down that path...

4. This proposal conflates window size with a decompression's overall
memory budget. For existing compressors, these two figures are very
close to one another, but in the future they might diverge. Would it be
better to specify a more holistic total memory limitation?

5. Is this sensible overall? Does anyone have strong feelings for or
against a mechanism like this?

Thanks,
Felix

[0] https://datatracker.ietf.org/doc/html/rfc7932#section-2
[1] https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.1.2
[2] https://github.com/facebook/zstd/blob/v1.5.0/lib/zstd.h#L1114-L1116
[3] https://github.com/facebook/zstd/blob/v1.5.0/lib/zstd.h#L1137-L1141
Received on Wednesday, 4 August 2021 21:47:14 UTC

This archive was generated by hypermail 2.4.0 : Wednesday, 4 August 2021 21:47:15 UTC