Re: Proposal: Adopt State Synchronization into HTTPbis

Josh Cohen and I are considering publishing a new draft on Subscriptions 
in HTTP, and as we think through the big design decisions, I ran across 
this excellent question from Watson Ladd bringing up the most 
fundamental question of them all:

On 11/6/23 4:47 PM, Watson Ladd wrote:
> On Tue, Oct 31, 2023 at 7:12 PM Michael Toomim<toomim@gmail.com>  wrote:
>> At IETF 118 I will present a proposal to adopt State Synchronization work into HTTPbis:
>>
>> Braid-HTTP:https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http  [1]
> <...snip...>
>
> The big sticking point for me is subscriptions. This is a deviation
> from the request/response paradigm that goes pretty deep into how
> clients and servers are coded and the libraries they use. It can of
> course be stuck on top of WebTransport, which might be the right way
> to do it, but then doesn't integrate with the other three parts.
>
> You might be better trying to layer this on top of HTTP and
> WebTransport, as ugly as that can be with regard to what
> intermediaries can do in order to get it into the hands of people
> faster, but if there's some strong reason not to do that I'm all ears.

Watson raises a basic choice in designing Subscriptions:

 1. Do we dare extend the basic request/response model to allow
    long-lived subscriptions?
 2. Or are subscriptions better layered on top— inside a WebSocket, or
    WebTransport connection?

I argue that (1) is actually a *much* better choice. Yes, it is 
fundamental. It extends the basic architecture of HTTP (hat tip to Roy 
Fielding) by extending REST into RESS (REpresentational State 
Synchronization). It adds a new basic feature to HTTP— the ability to 
subscribe to any resource, and get notified of its changes over time; 
throughout the entire web and HTTP ecosystem. Clients will stop guessing 
whether to reload cache, and will stop making redundant requests. 
Servers—which *authoritatively know* when resources change—will promise 
to tell clients, automatically, and optimally. Terabytes of bandwidth 
will be saved. Millions of lines of cache invalidation logic will be 
eliminated. Quadrillions of dirty-cache bugs will disappear. In time, 
web browser "reload" buttons will become obsolete, across the face of earth.

The alternative (2) is to add subscriptions on top of a WebSocket or 
WebTransport API, separate from HTTP resource semantics. But then HTTP 
resources themselves will not be subscribable. Programmers will be 
annoyed, and limit their use of HTTP to bootstrapping the initial HTML, 
CSS and Javascript; migrating all the interesting state onto this 
separate WebSocket or WebTransport mechanism, which will then require 
more and more of HTTP's features being added back into it: like (a) 
being able to GET the state, and also PUT changes to the it, over (b) 
multiple content-types (e.g. text/json, text/html, and image/png), while 
(c) supporting various PATCH types, across (d) a full-featured network 
of Proxies, Caches, and CDNs to scale the network. In conclusion, choice 
(2) leads to reinventing HTTP, within a WebSocket/Transport... on top of 
HTTP.

The clear way forward is subscribing directly to HTTP and REST state. An 
elegant way to see this is extending Request/Response into 
Request/Multiresponse. The Subscription can be a Request that receives 
multiple Responses, one for each update to the resource. There are many 
ways to format a Multiresponse; here's a straightforward and 
backwards-compatible option:

           Request:

              GET /chat
              Subscribe: timeout=10s

           Response:

              HTTP/1.1 104 Multiresponse
              Subscribe: timeout=10s
              Current-Version: "3"

              HTTP/1.1 200 OK
              Version: "2"
              Parents: "1a", "1b"
              Content-Type: application/json
              Content-Length: 64

              [{"text": "Hi, everyone!",
                "author": {"link": "/user/tommy"}}]

              HTTP/1.1 200 OK
              Version: "3"
              Parents: "2"
              Content-Type: application/json
              Merge-Type: sync9
              Content-Length: 117

              [{"text": "Hi, everyone!",
                "author": {"link": "/user/tommy"}}
               {"text": "Yo!",
                "author": {"link": "/user/yobot"}]

This is backwards-compatible because it encodes multiple responses into 
a regular response body that naïve intermediaries will just pass along 
blindly, like SSE. But upgraded H2 and H3 implementations can have 
native headers & body frames that repeat. It's all quite elegant. It 
fits right into HTTP. It feels as if HTTP was designed to make it possible.

We can add subscriptions to the basic fabric of HTTP, and free 
application programmers from having to write cache-invalidation logic. 
This will (a) eliminate bugs and code complexity; while simultaneously 
(b) improving performance across the internet, and (c) giving end-users 
the functionality of a realtime web by default. This is a fundamental 
change, but it is overwhelming beneficial. Then we can update Roy's 
dissertation. It's a good one, and deserves our care.

Michael