[w3c/manifest] Allow manifest processing to be invoked without going through an HTML document (Issue #1068)

Currently, per the spec, manifests can **only** be fetched via a `<link rel="manifest">` element in an HTML document. This is (as of #894 — July 2020) tightly coupled into the manifest spec, as the "processing a manifest" algorithm takes an HTML `link` element and a `Response` object as inputs. So the only way to invoke these steps is to have an open HTML document with a `link` to the manifest, and fetch the manifest using the HTML fetch machinery, and have a corresponding `Response` object.

However, this excludes or complicates a number of key use cases (which are in use today in various browsers), which fall under the umbrella of installing a web app when you don't already have the web page open. Use cases include:

- Installation directly from a "store" page (e.g. an app listing site or app, has a button to directly install a web app without first navigating to the app in a browser tab).
- Installation from another site (a possible future API where an app installation can be triggered from a third party origin).
- Automatic installation via sync (the user manually installs an app on device X, it syncs across to device Y and automatically gets installed without having to load the page on device Y).
- Automatic installation via admin policy (apps get installed when the user first opens their device).
- Automatic installation of a hard-coded set of default apps for a device or browser.

At the moment, browsers implement these features using a fairly clunky hack: they record a document URL. At install time, the browser creates an invisible web contents, loads the full HTML page, then triggers the app install, as required by the spec. There are a number of major downsides to this process:

- Having to load a full HTML page just to trigger the install then discarding it wastes a lot of bandwidth and time.
- Loading a page on startup without user consent is seen as a privacy issue. The user agent is sending requests to a third-party website without consent. (Granted, we would still have to send a request to fetch the manifest and icons, but these could be cached by the user agent ahead of time, e.g. the full text of these could be sent through the sync system.)
- It is necessarily flaky. Since pages can take an arbitrary amount of time to load the `<link>` element, we need to wait an arbitrary amount of time. In practice, we have a long (~1 minute) timeout, which means installations can fail because of network slowness (whereas if we directly loaded the manifest, we could just continue to wait as long as the connection remains open).

This hack even extends to developers and administrators. To get around the timing issue above, we have suggested that developers create a hidden "install page" - a blank HTML page that simply contains a `<link>` to the manifest, just so we use that as the document URL and avoid loading a full website. Then we have to tell administrators to use the "install page" instead of the regular start URL. These hacks create extra complexity for everyone.

Therefore, I propose that we:

1. Partially revert #894 to have "processing a manifest" take a document URL and manifest URL again (and let the HTML spec extract the URLs from the relevant objects and pass them in). This would allow the user agent to invoke "processing a manifest" from outside the HTML document.
2. Explicitly state (in normative text) that the user agent can do this without loading a document. However, add a requirement that if the user agent does this, they must in some way verify that there is a links back from the document to the manifest (otherwise a manifest on one origin could apply unauthorized metadata to app pages on someone else's origin).

Note: I am not currently proposing that we remove the document URL (which is what I proposed in #668, and could be a follow-up step). I am proposing that we remove the requirement that the document is _loaded_ in order to process the manifest. (This way, the user agent still has to verify the bidirectional link between the two origins, but it can be done ahead of time, not at install time -- for example if you are using sync, it has already been verified on the other device.) So we still need a document URL for the time being.

Note: This is related to #668 but not identical (it can be considered a stepping stone to that more advanced and risky change).

Note for completeness that I originally wrote this up in [this doc](https://docs.google.com/document/d/1NUsoTWkWrPFFNnxR00r1NF-MXzq-_rjl2ldAhZm-TEw/edit#) but all relevant details appear above.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3c/manifest/issues/1068

You are receiving this because you are subscribed to this thread.

Message ID: <w3c/manifest/issues/1068@github.com>