Caching and combining terms from independently processed remote contexts

Hello,

I have what's probably a Bad Idea(tm) that could use some input. For context, my background and interest in JSON-LD comes from working on social web projects. This informs a lot of my current experiments, because the fedi is a bit of a wild west.

Using the Fediverse Schema Observatory (https://observatory.cyber.harvard.edu/) you can see all the various shapes of ActivityStreams messages that are exchanged on the fedi. Observatory only sees a very small part of the network and heavily skews towards Mastodon instances, but that still makes it useful as a baseline.

One thing you can spot fairly quickly is that the value of @context is a little erratic. There are about 105 different "shapes" of @context accounted for in Observatory. The variance mostly comes from the inclusion of a local context document map that adds differing amounts of prefixes and terms. Unfortunately Mastodon and Misskey popularised this approach, rather than using a remote context. Which prefixes and terms also varies by implementation and implementation version. Mastodon has its own, Misskey and its forks have a slew of them, Pleroma/Akkoma add an @language of "und" for some reason and on it goes.

This makes a naive approach of caching the output of context processing based on hashing the @context somewhat ineffective. Worse still, the Pleroma/Akkoma lineage of instances include a remote context that's hosted on the instance the message came from. This effectively introduces an instance-level cache-busting key. You now have to get clever with caching, potentially dealing with a bounded LFU or LRU cache etc. to make this effective. How many instances you federate with, what instance software and version they use and how often you end up communicating with them also affect how effective any caching strategy is. I want to avoid this kind of complexity in my processor, so I've been looking at doing some "semantic" caching instead.

The majority of the context processing cost in fedi software comes from remote contexts, the AS vocabulary and W3ID Security v1. AS accounts for 140+ terms alone, though only a small subset are used in practice. Security v1 adds another 40+. But both the AS and Security v1 vocabulary are "independent." No matter within which active context you process the terms, you end up with the same definitions. This leads to a potential solution where the AS and Security v1 contexts can be independently processed in an empty active context, for example at application startup, and have those definitions copied into active context instead of reprocessed each time.

As an example, take an @context of:
[
  "https://www.w3.org/ns/activitystreams",
  "https://w3id.org/security/v1",
  {
    "manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
    "sensitive": "as:sensitive",
    "Hashtag": "as:Hashtag",
    "quoteUrl": "as:quoteUrl",
    "fedibird": "http://fedibird.com/ns#",
    "quoteUri": "fedibird:quoteUri",
    "quote": {
      "@id": "https://w3id.org/fep/044f#quote",
      "@type": "@id"
    },
    "toot": "http://joinmastodon.org/ns#",
    "Emoji": "toot:Emoji",
    "featured": "toot:featured",
    "discoverable": "toot:discoverable",
    "indexable": "toot:indexable",
    "attributionDomains": {
      "@id": "toot:attributionDomains",
      "@type": "@id"
    },
    "schema": "http://schema.org# <http://schema.org/#>",
    "PropertyValue": "schema:PropertyValue",
    "value": "schema:value",
    "misskey": "https://misskey-hub.net/ns#",
    "_misskey_content": "misskey:_misskey_content",
    "_misskey_quote": "misskey:_misskey_quote",
    "_misskey_reaction": "misskey:_misskey_reaction",
    "_misskey_votes": "misskey:_misskey_votes",
    "_misskey_summary": "misskey:_misskey_summary",
    "_misskey_followedMessage": "misskey:_misskey_followedMessage",
    "_misskey_requireSigninToViewContents": "misskey:_misskey_requireSigninToViewContents",
    "_misskey_makeNotesFollowersOnlyBefore": "misskey:_misskey_makeNotesFollowersOnlyBefore",
    "_misskey_makeNotesHiddenBefore": "misskey:_misskey_makeNotesHiddenBefore",
    "_misskey_license": "misskey:_misskey_license",
    "freeText": {
      "@id": "misskey:freeText",
      "@type": "schema:text"
    },
    "isCat": "misskey:isCat",
    "firefish": "https://joinfirefish.org/ns#",
    "speakAsCat": "firefish:speakAsCat",
    "sharkey": "https://joinsharkey.org/ns#",
    "hideOnlineStatus": "sharkey:hideOnlineStatus",
    "backgroundUrl": "sharkey:backgroundUrl",
    "listenbrainz": "sharkey:listenbrainz",
    "enableRss": "sharkey:enableRss",
    "vcard": "http://www.w3.org/2006/vcard/ns#"
  }
]

For clarity, I've grabbed an extreme example here. Most of the maps won't have this many terms defined. For example, a Misskey instance wouldn't include the Sharkey prefix and terms. But pretty much all of them will have the additional AS terms and a subset of the Mastodon terms included (toot prefix and terms). The Security v1 context is typically only present on Update Person activity, since that's generally the only place we need terms describing cryptographic keys. There are also cases where the local context map and Security v1 remote swap order, contributing to the explosion of @context "shapes."

In this case, we can initialise an empty context, then copy all the terms from a pre-processed AS context into them. When we hit the Security v1 remote, we can do the same for terms from a pre-processed Security v1 context. The one caveat here is that protected term redefinition still needs to be taken into account, following Step 27 from 4.2 Create Term Definition. In practice, remote contexts do not override protected terms from other vocabularies.

What then remains is processing the map, which in this approach is done as usual, paying the full cost of context processing. Any default @language and @direction, or their nullification, also needs to propagate. This is not something remote contexts normally set so in practice it's a no-op.

The same goes for a potential value for @vocab. The AS context sets it to _:, other context don't seem to touch it. But it never affects context processing of remote contexts, since no term needs it for IRI expansion. You just need to end up with the right value at the end, before you start expansion.

Based on my understanding, this works correctly as long as a remote context is independent. My current definition of independent is:
* Any prefixes needed for IRI expansion are defined in the remote context and don't need prefixes from active context.
* IRI expansion is not affected by @vocab.
* Terms in a remote context do not rely on terms from active context.

What I'm wondering now is:
* Has anyone else tried this particular trick of caching a "partial" result of context processing and reusing it?
* Where do folks think this could go wrong? I haven't looked at @include or @import yet, and it seems that dragons may be hiding there. But remote contexts don't seem to use those.

Kind regards,

-- 
Daenney

Received on Sunday, 6 September 2026 08:32:42 UTC