Re: [whatwg/fetch] Add compression dictionary negotiation and decoding to the fetch processing model (Issue #1739)

pmeenan left a comment (whatwg/fetch#1739)

Sorry about dropping this for so long. Now that we have a fair bit of experience with it, there are going to be connections at a few different points in the processing model.

Some of the processing is best done above the http cache (storing the dictionary so that the freshness is updated when fetched from cache) and some of the processing is best done below the http cache (adding available-dictionary and accept-encoding headers so "vary" isn't invalidated, decompressing so cached responses aren't dependent on the dictionary).

The split that Chrome uses so that dictionary-compressed responses in the http cache aren't dependent on the dictionary still being available is a debatable architecture decision though. It makes it more robust to the loss of the dictionaries but it also results in storing uncompressed responses (which we may compress separately for on-disk storage). Some browsers might decide that all of the processing should happen above the http cache where the loss of a dictionary would result in a cache miss but otherwise the artifacts are stored in their dictionary-compressed form.

The challenge with that case is that dictionary-compressed responses usually have "vary: available-dictionary" so, if you aren't careful, you could always download things like scripts twice (when using dictionaries to delta-compress script updates). First with no dictionary where the response sets itself as a dictionary for "versions of this script" but then the same script will be fetched again because now there is an available-dictionary (the script itself).

My plan is to document what Chrome has implemented but there are other ways to solve the layering while still maintaining compatibility.

Here's the outline of the pieces I'm working on:

Add support for "link" fetching of "compression-dictionary"
- Mirror "preload" but with lowest priority
- Set value of Sec-Fetch-Dest request header to compression-dictionary
- Add compression-dictionary to the list of destinations


Add processing of Use-As-Dictionary response header (above http cache)
- Calculating expiration from cache-control/expires
- Validate URLPattern is same-origin
- Ignore unknown sf-dict keys
- Triple-keyed cache (top-level document site, frame origin, request origin) for dictionary entries


Add support for matching dictionaries (below http cache, above network)
- Restricted to requests that can possibly be CORS-readable (no matching for no-cors requests - yet)
- Restricted to requests were cookies or storage are allowed (as triple-keyed data)
- Triple-keyed cache lookup (top-level document site, frame origin, request origin)
- Collect all patterns that match the fetch destination and where the URLPattern matches then pick the one with the longest URLPattern string (or most recently fetched for identical-length patterns)
- If a match is found:
    - Add Available-Dictionary and Dictionary-ID request headers
    - Add dcb and dcz to Accept-Encoding request header


Handle dcb and dcz Content-Encoding's
- Below the disk cache so that cached responses do not also need the dictionary to be available
- Extract dictionary ID from response data stream and validate against requested Available-Dictionary
- Fail response if CORS-readability checks fail


Add dictionary cache clearing
- Manual as well as clear-site-data (either cookies or cache)

-- 
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/fetch/issues/1739#issuecomment-3206883769
You are receiving this because you are subscribed to this thread.

Message ID: <whatwg/fetch/issues/1739/3206883769@github.com>