Re: Compression dictionary draft ID - draft-meenan-httpbis-compression-dictionary-00

There are many possible architectures that can work of course, but the simplest one and most generic one is to hope the dictionary is present in your local cache and fetch it (with Cache-Control: only-if-cached; If-Match: etag).

I would definitely avoid using a KV for a generic solution, maybe in the far future for high value use case.

I can't simply store an arbitrary number of dictionaries in KV, this can very well explode, and some intelligent eviction is needed.

We also don't want to store multiple versions of the same resource compressed with different algorithms, even for statically compressed assets. It consumes valuable space that could be used to cache other assets instead.

In any case, having the URL and ETag (and obviously ETags are not always present either) allows us to have *something* in production almost out of the box, having just the cache requires building new stuff, which is a much higher bar for entrance.

Best,
Vlad

> On Aug 8, 2023, at 11:46 AM, Patrick Meenan <patmeenan@gmail.com> wrote:
> 
> That would necessarily couple the use of a dictionary with the negotiation of the storage. They way it is currently set up, "use-as-dictionary" is one way to populate the dictionary but the dictionaries could also be pre-loaded into a client or otherwise negotiated.
> 
> As far as state, if you're passively observing traffic to compress and not actively managing the dictionaries, doesn't something like this work?
> 
> 1. See response with "use-as-dictionary" response header
> 2. Store dictionary in key-value store, using the hash of the payload as the key
> 3. See request with "sec-available-dictionary" request header
> 
> For supporting static, offline resource compression:
> - If a version of the resource is in cache compressed with the requested dictionary, serve it
> - If not, kick off an async task to compress the cached full response with the dictionary (if one is available in KV)
> 
> For dynamic responses:
> - If the dictionary is in KV, use it to compress the response
> 
> The main "state" that needs to be managed is the storage of the dictionary keys, indexed by the appropriate hash.
> 
> If the CDN is generating the dictionaries (or adding the headers) there's not much more state than that either.
> 
> I'm sure I'm probably hand-waving a lot of things that need to be done at scale when handling thousands of dictionaries and thousands of simultaneous requests for the same dictionary but I'd hate to add complexity to the negotiation itself if it isn't absolutely necessary.
> 
> On Tue, Aug 8, 2023 at 9:49 AM Vlad Krasnov <vlad@cloudflare.com <mailto:vlad@cloudflare.com>> wrote:
>> I think it would make the proposal easier to support from a CDN POV if the Sec-Available-Dictionary included the dictionary URL and Etag, otherwise it requires too much state keeping.
>> 
>> Given a URL and an Etag it is pretty easy to have a best effort dictionary compression.
>> 
>> Best,
>> Vlad