Re: Compression dictionary draft ID - draft-meenan-httpbis-compression-dictionary-00

One area I could see benefitting from reducing some overhead would be
around the hash of the dictionary contents since it could be expensive to
calculate.  One thing we could do there is have an additional (optional)
field in the use-as-dictionary response where the origin could populate the
hash as "hint" so caches or clients could use the advertised hash to
determine if it needs to store the dictionary or if it already has it.  The
language would have to be careful to require that the actual hash still be
calculated before storing (or use) to be sure that the has represents the
contents accurately but the hint could make the processing lighter weight,
particularly in the CDN flow.

On Tue, Aug 8, 2023 at 11:46 AM Patrick Meenan <patmeenan@gmail.com> wrote:

> That would necessarily couple the use of a dictionary with the negotiation
> of the storage. They way it is currently set up, "use-as-dictionary" is one
> way to populate the dictionary but the dictionaries could also be
> pre-loaded into a client or otherwise negotiated.
>
> As far as state, if you're passively observing traffic to compress and not
> actively managing the dictionaries, doesn't something like this work?
>
> 1. See response with "use-as-dictionary" response header
> 2. Store dictionary in key-value store, using the hash of the payload as
> the key
> 3. See request with "sec-available-dictionary" request header
>
> For supporting static, offline resource compression:
> - If a version of the resource is in cache compressed with the requested
> dictionary, serve it
> - If not, kick off an async task to compress the cached full response with
> the dictionary (if one is available in KV)
>
> For dynamic responses:
> - If the dictionary is in KV, use it to compress the response
>
> The main "state" that needs to be managed is the storage of the dictionary
> keys, indexed by the appropriate hash.
>
> If the CDN is generating the dictionaries (or adding the headers) there's
> not much more state than that either.
>
> I'm sure I'm probably hand-waving a lot of things that need to be done at
> scale when handling thousands of dictionaries and thousands of simultaneous
> requests for the same dictionary but I'd hate to add complexity to the
> negotiation itself if it isn't absolutely necessary.
>
> On Tue, Aug 8, 2023 at 9:49 AM Vlad Krasnov <vlad@cloudflare.com> wrote:
>
>> I think it would make the proposal easier to support from a CDN POV if
>> the Sec-Available-Dictionary included the dictionary URL and Etag,
>> otherwise it requires too much state keeping.
>>
>> Given a URL and an Etag it is pretty easy to have a best effort
>> dictionary compression.
>>
>> Best,
>> Vlad
>>
>