RE: Mnot's Pub/Sub for the Web

This is a very interesting space, and I’m glad we have two such solid contenders.  I’m not convinced this fits squarely within HTTP’s mandate, as this seems more like a protocol on top of HTTP than a pure extension to HTTP.  Perhaps like OHAI, there might be enough interest to warrant a dedicated working group?

From: Kévin Dunglas <kevin@dunglas.fr>
Sent: Tuesday, February 22, 2022 12:56 PM
To: Michael Toomim <toomim@gmail.com>
Cc: HTTP Working Group <ietf-http-wg@w3.org>; Mark Nottingham <mnot@mnot.net>
Subject: Re: Mnot's Pub/Sub for the Web

Thanks for bringing this topic to the list again!

On the Mercure side, the spec has stabilized. Several open-source and proprietary implementations are available (https://mercure.rocks/spec#implementation-status), and adoption is growing: 2.7K stars on GitHub, dozens of open source projects using it, large companies publicly declaring use...

Many new use cases have been reported on the bug tracker over the years, and we improved the spec to cover most of them. Some minor issues still need to be handled (https://github.com/dunglas/mercure/labels/spec), but we're very soon to publish the final version of the specification.

As demonstrated by the discussions on Hacker News, Mark's great article, and by the adoption of Mercure, the community is in demand of a pub/sub standard for web resources.

Even if most discussions occurred on GitHub, Slack, Twitter, and other channels instead of on IETF mailing lists, the Mercure spec is now implemented by production-grade "running code" and has reached "rough consensus".

Mercure is less ambitious than Braid. Its scope is more limited. It is focusing on providing a simple pub/sub protocol for web content proved working with the current web infrastructure (web browsers, proxies,, firewalls, etc). In its current state, it doesn't require any JS library or polyfill client-side.

The spec is very similar to the WebSub specification from the W3C, but mainly targets web browsers instead of servers. As WebSub, Mercure uses a hub to distribute web resources, which allows implementing the protocol easily even in legacy applications, with languages not designed to handle long-living connections (e.g. PHP), and when using modern infrastructure such as serverless and edge computing platforms (https://dunglas.fr/2019/07/mercure-real-time-apis-for-serverless-and-beyond/). Unlike WebSub, Mercure natively supports authorization, end-to-end encryption, and state reconciliation. Both clients and servers can be publishers.

Currently, Mercure only allows using SSE as transport, but we'll maybe allow using other transports such as WebSockets and Web Transports, probably as extensions to the current spec, to cover use case such as transmitting non-base64-encoded binary data (https://github.com/dunglas/mercure/issues/616).

Braid is very interesting and has a much broader scope (state synchronization, P2P, etc). It also requires more changes to the current software stack to be natively supported by the web platform. Mercure overlaps only with the "subscribe" feature of Braid, and I've the feeling than Braid could use Mercure (and probably WebSub too) for its subscribe feature, at least in a first iteration.

I wonder how we can move forward regarding the standardization of a pub/sub protocol for web content and web browsers. Even if Mercure gained traction outside of the IETF, it hasn't on this group. I was thinking about proposing the final version of the spec as an independent-track RFC, or to the W3C as it is very close to WebSub, and is also related to the other specs published by the Social Web Working Group (ActivityPub, and even Solid). But as the this topic is discussed again, maybe could we work on a pub/sub protocol here?



On Sun, Feb 20, 2022 at 10:39 AM Michael Toomim <toomim@gmail.com<mailto:toomim@gmail.com>> wrote:

Hello, HTTP!

Today Mark Nottingham posted a great articulation of the issues programmers face when choosing between using SSE, WebSockets, and WebTransports:

https://www.mnot.net/blog/2022/02/20/websockets


I'll attempt to summarize Mark's beautiful insight as: in almost all cases, what the programmer *really* wants is a Pub/Sub protocol, not an arbitrary socket. And we could standardize a Pub/Sub protocol, and that would have great benefits.

These benefits are real and I think could improve performance dramatically. CDNs could cache realtime updates, not just static data.

However, I'll take Mnot one further, and propose that when a programmer is choosing a Pub/Sub protocol, what he *really* wants is a State Synchronization protocol, not an arbitrary Pub/Sub protocol.

He wants to Subscribe specifically to *state updates*. He wants to Publish specifically *updates to state*.

What we need is not a general Pub/Sub standard, but specifically a State Synchronization standard. State Synchronization is a constrained type of general Pub/Sub. And we'll need to constrain Pub/Sub in this way to address some of the issues Mark brings up, such as:

> There are also some architectural/philosophical concerns about how non-final responses **relate to the state of the resource**.

The relationship between a server's "responses" and the "state of the resource" is what a State Synchronization protocol defines. And, in fact, we have two proposed solutions to State Synchronization in the IETF!

Braid:         https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-braid-http

Mercure:    https://datatracker.ietf.org/doc/draft-dunglas-mercure/


I am seeing a growing awareness that HTTP needs to add State Synchronization abilities, as well as excitement about the new fundamental power it gives programmers on the web.

These protocols transform HTTP from a State *Transfer* into a State *Synchronization* protocol. Whereas a transfer protocol can move a resource from server to client in a single request/response, it requires an application programmer to take over if the resource ever changes after the response completes. That sucks for programmers. A synchronization protocol provides a much better programming abstraction. The programmer just says "I want state X", and can assume it will be kept up-to-date by the protocol.

If we standardize this, we also get CDNs that automatically cache dynamic content (the stuff currently hidden within websockets), just as easily as they cache static content today. We get collaborative editing and offline modes available in web apps for free. We also take an important step towards decentralizing the web, by creating an open standard for the trickiest part of decentralized app development — data synchronization — that is compatible with P2P CRDT and OT algorithms.

Since this all seems to be coming together, I would like to know what HTTPbis as a group thinks. Is there interest in this topic?

If so, what aspects might we want to work on?