Re: Negotiating Window Limits for Content Encodings

Hello Felix,

On Wed, Aug 04, 2021 at 05:47:00PM -0400, W. Felix Handte wrote:
> Hello all,
> 
> I hope you had a pleasant IETF 111.
> 
> TL;DR:
> 
> I want to specify an option along these lines:
> 
>   Accept-Encoding: zstd;w=18
> 
> Which in this case would mean that a client will accept a
> zstd-compressed response, only if the window size is less than or equal
> to 256KB (2 ** 18 bytes).
(...)

I personally think it's an excellent idea, though I'm really not sure
about its effectiveness because:
  - probably that some intermediaries or servers will not match
    "zstd;w=18" against any known token ;

  - some servers or intermediaries might simply ignore "w=18" for quite
    some time ;

  - other intermediaries might possibly transcode this to "zstd" before
    passing it on to the next hop, especially if they try to reorder
    fields following a different preference order ;

  - in any case the client will need to be prepared to reject the
    response if it detects that expectations are not met.

However, I like the idea because it would allow a client to send for
example:

   Accept-Encoding: zstd;w=18, gzip

And at least for servers that don't recognize the first one as a token,
the second one will naturally be used. And servers which understand it
could decide that they prefer some other algos below certain window
sizes if it costs less for them due to implementation constraints.

Also, it will be needed to state that the server should ignore too low
values so that it doesn't degrade a cache's contents (i.e. zstd;w=0).
And for caches it is probably less easy to define what is compatible
with what. Maybe you'll need to suggest the same thing in the response.

> Some of those Questions:
> 
> 1. Is this worth making a generic mechanism and specifying it as an
> option for any content-coding for which it's meaningful? Or should I
> just restrict this to Zstd, which is where I'm aware of the most need?

Given what you said about Brotli, it seems that it should be generic,
though it's not certain that such an argument easily applies to all
algorithms.

> 2. Is this the best place to insert such a mechanism? Or should it be
> its own header? (Would that be easier for caches to handle?)

I think that having it in a different header field would make it difficult
to deal with lists (e.g. "zstd;w=18,gzip" hardly transcodes to multiple
headers).

> 3. Is it effective to tack on an option like this to existing
> content-coding identifiers? Since the existing installed base does not
> produce or recognize this option, that means it can only ever be best
> effort for existing content-codings. A heavy-weight alternative would be
> to introduce new identifiers that mandate handling this signal, but I
> don't want to go down that path...

There's the risk that everyone needs a different value :-/  However it
could still be a valid alternative to imagine 3 profiles for each algo,
"low-memory" for inline transcoding (e.g. less than 64k), "mobile" (e.g.
less than 1 MB) and the default, unlimited. But even with this I'm pretty
sure that everyone will have a different opinion.

> 4. This proposal conflates window size with a decompression's overall
> memory budget. For existing compressors, these two figures are very
> close to one another, but in the future they might diverge. Would it be
> better to specify a more holistic total memory limitation?

As long as the relation between the two is understood for each algo,
it's the client's responsibility to advertise the correct value based
on its capabilities. An alternative could be to pass an argument "l"
for "level" instead of window, which would correspond to the max
compression level for the algo, given that many algos naturally
support such an argument.

> 5. Is this sensible overall? Does anyone have strong feelings for or
> against a mechanism like this?

I do like it but am unsure about the best path to make it seamless to
integrate. I wouldn't be surprised that you end up proposing that
clients send something like "zstd;w=18, zstd" to be sure that it's at
least recognized. But I think it's worth studying existing deployments
to figure the best path forward.

Regards,
Willy

Received on Thursday, 5 August 2021 02:39:03 UTC