Re: Background sync & push messaging: declarative vs imperative

Apologies, that e-mail was sent prematurely by accident, my final comment
was going to be:

My only other concern is around this *"*we could either silently throttle
precisely scheduled tasks if it turns out they’re not showing notifications".
There will be scenarios where this will be desired, expected and valid
behaviour to have a notification at a specific time to perform some
background processing, so why are we giving the user something to nag them
that work is going on. It's not so much of a background task, but rather
changing the UI from DOM to a notification.
While I think of it, I don't see any discussion of how a developer can
remove a notification, for example if the data is no longer of any use.

Regards,
Matt


On Fri, Dec 20, 2013 at 2:34 PM, Matt Gaunt <mattgaunt@google.com> wrote:

> For developers I think the imperative approach would be easier to grasp
> and more flexible. The declarative approach can probably do everything that
> the imperative API could do, but I fear it will end up with developers
> hacking around the limitations of using the declarative API of push
> notifications and caches files to handle info when the page is open.
>
> The two approaches outlined here seem comparable to AppCache and
> ServiceWorker.
>
> While security is a concern with the imperative approach, I can't see how
> an attacker could run a background service without the user visiting their
> site.....
>
> A permission pop-up of some sort should be the first barrier to setting up
> push-notifications and background services.
>
> Requiring a user-interface for un-throttled push notifications seems
> unusual, I have no problem flagging bad behaviour to a user (i.e. this site
> is sending X number of pushes a minute), but there may be scenarios where
> the users expects and wants that behaviour.
>
> My only other concern is around this *"**we could either silently
> throttle precisely scheduled tasks if it turns out they’re not showing
> notifications"* I
>
> Regards,
> Matt
>
>
>
> On Thu, Dec 19, 2013 at 9:32 PM, Maciej Stachowiak <mjs@apple.com> wrote:
>
>>
>> On Dec 19, 2013, at 9:02 AM, John Mellor <johnme@google.com> wrote:
>>
>> [cross-posted to public-webapps and public-sysapps]
>>
>> A couple of us from Chrome have taken a holistic look at how we could add
>> standardized APIs for web apps to execute/sync in the background.
>>
>> This is an important capability, yet can be safely granted to
>> low-privilege web apps, as long as battery consumption and metered data
>> usage are (very) carefully limited.
>>
>>
>> Running arbitrary JS in the background, without the relevant page open,
>> and with full networking capabilities, does not seem safe to me. How did
>> you conclude that this capability can safely be granted to low-privilege
>> web apps?
>>
>> Some of the specific threats I'd be concerned about include:
>>
>> (1) An attacker could use this capability to spread a botnet that can be
>> used to mount DDOS attacks or to perform offline distributed computations
>> that are not in the user's interest, without needing a code execution
>> exploit or sandbox escape.
>>
>> (2) An attacker could use this capability to widely spread an innocuous
>> looking background service service which, based on remote
>> command-and-control, attempts zero-day exploits against the user's browser
>> at a time of the attacker's choosing. This can happen without visiting the
>> malicious site, indeed, while a user doesn't have a browser open, and can
>> possibly happen faster than even the fastest-updating browser could
>> plausibly patch.
>>
>> These both seem unsafe to me, compared to the security of the Web as it
>> stands today. And I don't think a permissions dialog would be a sufficient
>> mitigation, since it is hard to explain the danger to users.
>>
>> Your semi-declarative proposal seems potentially less dangerous, though I
>> haven't thought through all of the risks in detail yet.
>>
>> Regards,
>> Maciej
>>
>>
>> This is not a spec proposal - instead I’m looking for feedback on the
>> high-level choice between an imperative or declarative set of APIs.
>>
>>
>> USE CASES
>>
>> 1. Sync when next online
>> - Need to send emails & document edits written offline as soon as the
>> device goes back online (even if app is not open), and get back sending
>> status, or for documents the result of merging the user's diff with any
>> changes made by other users.
>> e.g. document editing / email client / instant messaging / play-by-mail
>> games
>>
>> 2. Background sync
>> - Need to periodically fetch new content in the background, so user will
>> have fresh content if they open the app offline.
>> - Content often depends on device location (hence requires polling).
>> e.g. news / weather / maps / social stream
>>
>> 3. Large background transfers
>> - Need to upload/download large files. App won't remain in the foreground
>> for long enough, so this must happen in the background; and even in the
>> background, the app is unlikely to be allowed to stay open long enough
>> (e.g. iOS 7 limits the execution time of Background Fetch to 30 seconds<http://www.objc.io/issue-5/multitasking.html>),
>> so there needs to be a way of handing over large transfers to the
>> browser/OS to perform efficiently without the app having to keep running..
>> - Ideally such transfers would auto-resume after device reboots.
>> e.g. file syncing / movies / video camera
>>
>> 4. Push notifications
>> - Need instant push notification of messages received by the server, or
>> as soon as the device comes online if it was offline (even if app is not
>> open).
>> e.g. email client / instant messaging / play-by-mail games
>>
>> 5. Push to sync (tickle)
>> - Sometimes a push needs to trigger a sync, for example to fetch email
>> attachments.
>> e.g. file & document syncing / email client / play-by-mail games
>>
>> 6. Delayed local notifications
>> - Need to show notifications at some time in the future.
>> - Various possible interactions with timezone changes and daylight
>> savings time.
>> e.g. egg timer / alarm clock / calendar
>>
>> 7. Delayed remote notifications
>> - Consider a calendar app: it must reliably show notifications for
>> remotely-added events, even if the app hasn’t been opened since the event
>> was added. Hence there must be some mechanism for a push message from the
>> server to cause a local notification to be set at some time in the future.
>> e.g. calendar / cloud synced alarm clock
>>
>>
>> Solving all of these requires a variety of different APIs. But it's
>> important that they fit together well, and it turns out that the API design
>> choices are closely interrelated.
>>
>> There are two broad approaches to providing such capabilities on the web:
>>
>>
>> A) IMPERATIVE APPROACH
>>
>> One approach is to allow JavaScript to be run on demand in the
>> background. The prime candidate for this is to extend ServiceWorkers<https://github.com/slightlyoff/ServiceWorker/blob/master/explainer.md>to be a generic mechanism for receiving events in the background (without a
>> page having to be open).
>>
>> 1. Sync when next online
>>
>> To address use case #1, add an API that requests your ServiceWorker to
>> get woken up once in the background, as soon as possible after the device
>> next goes online. This could be an extension of the Task Scheduler API<http://www.w3.org/2012/sysapps/web-alarms/>.
>> This API should probably only be available when running in the foreground.
>>
>> navigator.taskScheduler.addOneShot({
>>     requireOnline: true
>> }, myData);
>>
>> 2. Background sync
>>
>> Similarly, to address use case #2, let web apps request their
>> ServiceWorker to be periodically run in the background, at an interval of
>> the UA’s choice (using aggressive heuristics<http://lists.w3.org/Archives/Public/public-sysapps/2013Nov/0039.html>to choose the most battery/data-efficient moment to run it); though we
>> could allow the developer to specify a minimum interval (i.e. if content
>> updates daily, you could specify that there’s no point syncing more often).
>>
>> navigator.taskScheduler.addRepeating({
>>     minInterval: "2h"  // UA will pick a >= interval
>> }, myData);
>>
>> We’d probably strictly limit the execution time of the ServiceWorker to
>> conserve battery (by comparison, in native apps iOS 7 limits the
>> execution time of Background Fetch to 30 seconds<http://www.objc.io/issue-5/multitasking.html>
>> ).
>>
>> 3. Large background transfers
>>
>> Although we’d be limiting background execution time of the ServiceWorker,
>> we can compensate by making it possible to initiate long-running
>> up/downloads, managed by the browser/OS.
>>
>> This could be done by letting you mark async XMLHttpRequests as
>> "persistent", somewhat like the Beacon API <http://www.w3.org/TR/beacon/>,
>> except that the browser would periodically retry requests if the device was
>> offline, and make the results available to the page or ServiceWorker next
>> time the web app is launched.
>>
>> Alternatively we could reuse ServiceWorker’s scriptable Cache objects<https://github.com/slightlyoff/ServiceWorker/blob/master/caching.md>.
>> You’d get syntax something like this:
>>
>> caches.set("movie-cache", new Cache(
>>     "http://my-app.com/The%20Lion%20King.mp4",
>>     "http://my-app.com/The%20Jungle%20Book.mp4"
>> ));
>>
>> This would cause the UA to download these large files in the background.
>> Once all the downloads in the cache are finished, they would show up in
>> caches.get("movie-cache").
>>
>> 4. Push notifications & 5. Push to sync (tickle)
>>
>> To address use cases #4 and #5, implement something like the Push API<http://www.w3.org/TR/push-api/>,
>> allowing servers to remotely wake up a ServiceWorker. The ServiceWorker can
>> then sync, show a notification, etc. (For battery life reasons, silent push
>> messages probably need to be throttled, in which case we could consider
>> adding a never-throttled variant for messages that show a user-visible
>> notification).
>>
>> PUT /push/send/device-c0fa407591 HTTP/1.1
>> Host: browser-push-server.com
>>
>> version=5
>>
>> 6. Delayed local notifications
>>
>> To address use case #6, extend the Notifications API<http://notifications.spec.whatwg.org/>so notifications can exist independently of the lifetime of a web page, and
>> when activated will wake up the ServiceWorker or web page and fire an event
>> on them.
>>
>> There also needs to be a way to fire a notification after a precise time
>> delay; the natural way to do this would be to also allow exact scheduling
>> of time sensitive tasks using the Task Scheduler API<http://www.w3.org/2012/sysapps/web-alarms/>,
>> but for battery reasons we shouldn’t give web apps carte blanche here;
>> instead to prevent abuse we might want to only allow exact scheduling if
>> you also show a user-visible notification. So we could either silently
>> throttle precisely scheduled tasks if it turns out they’re not showing
>> notifications, or we could not implement exact scheduling in Task Scheduler
>> at all, and instead extend Notifications to have an optional time delay.
>>
>> new Notification("Meeting about to begin", {
>>     utcTime: 1386011460303,
>>     body: "Room 101",
>>     launchUrl:  "https://my-app.com/?from=notification"
>> });
>>
>> 7. Delayed remote notifications
>>
>> By combining the APIs from the previous two sections, you can have a push
>> message handler that schedules a delayed notification, satisfying use case
>> #7.
>>
>>
>> B) (SEMI-)DECLARATIVE APPROACH
>>
>> Alternatively, it would be possible to solve almost all these use cases
>> without having to run JavaScript in the background, with a combination of
>> declarative APIs. This seems to be a new idea, so it warrants more detailed
>> explanation.
>>
>> A key premise of this approach, is that it’s always ok to wake up the
>> device/app in order to display a user-visible notification, since these
>> provide value to the user (or if they don’t, the user can deal with it by
>> revoking the permissions of the web app). But other than that, it’s best to
>> limit battery/bandwidth/RAM consumption in the background.
>>
>> *These are just pseudo-APIs to demonstrate possible capabilities and not
>> actual proposals.*
>>
>> 1. Sync when next online & 3. Large background transfers
>>
>> For use cases #1 and #3, uploads and downloads need to happen in the
>> background, and in a declarative world these would need to be fully managed
>> by the browser. As in the imperative approach to use case #3, we could
>> reuse ServiceWorker’s scriptable Cache objects<https://github.com/slightlyoff/ServiceWorker/blob/master/caching.md>for background downloads, either from a ServiceWorker, or by exposing them
>> to ordinary web pages:
>>
>> caches.set("movie-cache", new Cache(
>>     "http://my-app.com/The%20Lion%20King.mp4",
>>     "http://my-app.com/The%20Jungle%20Book.mp4"
>> ));
>>
>> This would cause the UA to download these large files in the background.
>> Once all the downloads in the cache are finished, they would show up in
>> caches.get("movie-cache").
>>
>> Similarly background uploads are sometimes necessary. For use case #1
>> (e.g. email), it is important that the background upload happen as soon as
>> possible once the device goes back online; for use case #3 (e.g. video
>> sharing), the background uploads need to support uploading large files,
>> however timeliness is less important, and it might be best for the upload
>> to only happen over WiFi. It might still be possible for the two to use the
>> same syntax; for example we could allow adding<https://github.com/slightlyoff/ServiceWorker/issues/118>a
>> Request<https://github.com/slightlyoff/ServiceWorker/blob/062ecbc967e11969adef85fd044a3fab0cdf7e1c/service_worker.ts#L210>object to a Cache instead of just URLs:
>>
>> caches.set("outbox", new Cache(new Request({
>>     method: "POST",
>>     url: "http://my-app.com/send-mail",
>>     body: my_mime_multipart_message
>> }), ...));
>>
>> The UA would perform these requests in the background, and store the
>> response in the cache object. As for urgency, perhaps there could be some
>> "urgent_request_by_user" flag that the web developer can set to indicate
>> that the user explicitly requested this and intends it to be sent over
>> cellular data.
>>
>> 2. Background sync
>>
>> The mechanism above is powerful, but sometimes instead of a one-shot
>> up/download you need to sync data that regularly updates.
>>
>> Push can be great for that; but sometimes you have to poll, for example
>> weather/map apps syncing forecasts/tiles based on your current location.
>> And sometimes (though perhaps rarely) even when you can push it’s more
>> efficient to poll smartly - for example if my social network feed has new
>> posts several times a minute, I really don’t want it waking up my phone
>> radio every few minutes while I’m asleep (especially if I forgot to plug it
>> into a charger).
>>
>> We could extend the caches mechanism above with the ability for the UA to
>> periodically check -- in the background -- for updates to the files in the
>> cache (using standard HTTP caching logic). The UA could optionally include
>> the user’s geolocation in a header when doing so.
>>
>> caches.set("sync-cache", new Cache(url1, url2, ...));
>> caches.requestBackgroundSyncing("sync-cache", {
>>     geolocation: true
>> });
>>
>> The client would indicate with cookies or somesuch any user preferences
>> about what should be synced. The server could update the cookies to
>> indicate how much it has synced, so during the next sync it knows what to
>> do.
>>
>> However this is still quite limiting, as it requires the client to know
>> in advance the URLs of the files the server will want it to download. It
>> would probably lead to hacks where clients request that a bunch of
>> meaningless foo1, foo2, foo3 urls get synced, and the actual content of
>> those urls gets rotated server-side. A more flexible approach would add
>> indirection, and let the server provide a barebones "manifest" file, that
>> would just be a newline-separated list of URLs to sync.
>>
>> caches.set("sync-cache", new CacheManifest("/sync-manifest"));
>> caches.requestBackgroundSyncing("sync-cache", {
>>     geolocation: true
>> });
>>
>> The UA would periodically check for updates to the manifest file, fetch
>> any new URLs, update existing ones, and delete cache entries for any URLs
>> removed from the manifest. Updates would presumably be atomic (sync all
>> files in the manifest before updating the cache exposed to the page). I’ve
>> glossed over various details, such as what constitutes an update to the
>> manifest file, but it should be possible to define something reasonable,
>> learning from the lessons of AppCache.
>>
>> As with the imperative approach, the UA would use various heuristics<http://lists.w3.org/Archives/Public/public-sysapps/2013Nov/0039.html>to determine when is a good time to sync each app, such as batching, screen
>> on, wifi available, charging, or even what times of day each web app is
>> typically used.
>>
>> 4. Push notifications
>>
>> For use case #4, there needs to be a standardized way for an app’s server
>> to instantly push user-visible notifications to the device (with automatic
>> retry if the device is offline). Interacting with the notification would
>> launch the app, at the given URL.
>>
>> POST /push/send/device-c0fa407591 HTTP/1.1
>> Host: browser-push-server.com
>> Content-Type: application/json
>>
>> {
>>     "title": "Message from Ben",
>>     "body":  "This should arrive instantly :)",
>>     "sound": "https://my-app.com/you-got-mail.mp3",
>>     "launchUrl":   "https://my-app.com/?from=notification"
>> }
>>
>> 5. Push to sync (tickle)
>>
>> In the push notification example above, the notification didn’t contain
>> any data other than the notification text, and the URL to launch when the
>> notification was clicked. It would be reasonable to add a "data" member to
>> the JSON, which would somehow be passed to the web page when the
>> notification is clicked.
>>
>> However most push messaging servers have strict size limits on the
>> payload (e.g. 4096 bytes). There are many cases, where an app needs to send
>> more than this, for example an email client that needs to download the
>> attachments that accompany an email, so that if the user clicks the
>> notification whilst offline, they will be able to view the attachments.
>>
>> To handle this case, we can allow a silent push message that kicks off an
>> immediate background sync of one or more of the named caches from "2.
>> Background sync" above:
>>
>> POST /push/send/device-c0fa407591 HTTP/1.1
>> Host: browser-push-server.com
>> Content-Type: application/json
>>
>> {
>>     "updateCaches": ["sync-cache", "attachments-cache"]
>> }
>>
>> 6. Delayed local notifications
>>
>> For use case #6, there needs to be a way to locally schedule
>> notifications with a time delay (since the device might be offline the
>> whole time, you can’t rely on push notifications for this). We could extend
>> the Notifications API <http://notifications.spec.whatwg.org/> as follows:
>>
>> new Notification("Meeting about to begin", {
>>     utcTime: 1386011460303,
>>     body: "Room 101",
>>     launchUrl:  "https://my-app.com/?from=notification"
>> });
>>
>> The notification would be delayed until the given moment. A URL is
>> provided to launch the app in response to the user clicking on the
>> notification, if the app isn’t already running; usually, this URL would be
>> available offline due to AppCache or a ServiceWorker<https://github.com/slightlyoff/ServiceWorker/blob/master/explainer.md>.
>> An API like Notification.get()<http://notifications.spec.whatwg.org/#dom-notification-get>would let you read and cancel pending notifications.
>>
>> 7. Delayed remote notifications
>>
>> A subtle variant of notifications is use case #7. If you add a same day
>> event to your cloud calendar with a reminder 10 minutes before the event,
>> then the device goes online for a while but the calendar app does not get
>> launched by the user, and the device goes back offline for the hours
>> leading up to the event, the calendar app needs to be able to fire that
>> notification 10 minutes before the event, despite (in this declarative
>> model) having been executed neither at the time the push notification
>> arrived, nor at any time since then.
>>
>> Since it doesn’t get executed, such an app can’t locally schedule a
>> delayed notification, so instead the push notification API introduced for
>> use case #4 could be extended so you can specify a time delay before the
>> notification fires (as with delayed local notifications).
>>
>> POST /push/send/device-c0fa407591 HTTP/1.1
>> Host: browser-push-server.com
>> Content-Type: application/json
>>
>> {
>>     "utcTime": 1386011460303,
>>     "title":   "Meeting about to begin",
>>     "body":    "Room 101",
>>     "launchUrl":     "https://my-app.com/?from=notification"
>> }
>>
>> We’d need to also support push messages that cancel earlier delayed push
>> notifications (e.g. if the event later gets removed from your cloud
>> calendar). And the same API that lets you inspect local delayed
>> notifications could also read/cancel pushed delayed notifications.
>>
>>
>> CONCLUSION
>>
>> The imperative approach (A) seems a cleaner set of APIs, from an extensible
>> web <http://extensiblewebmanifesto.org/> point of view.
>>
>> However whenever a ServiceWorker (or equivalent) runs in the background,
>> it requires a full JS interpreter, and all the associated browser machinery
>> to support APIs like Geolocation, which together consumes a significant
>> amount of RAM. This can be a problem on mobile; for example on Windows
>> Phone, background agents are restricted to 11 MB of RAM<http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202942(v=vs.105).aspx#BKMK_ConstraintsforallScheduledTaskTypes>on devices with less than 1GB, otherwise 20 MB, and are terminated if they
>> exceed this limit. This is presumably in order to ensure that the
>> foreground app (and other important processes) remain responsive (and don’t
>> get evicted). The limits on Android and iOS are vaguer and
>> device-dependent, but increased background RAM usage could potentially be a
>> deal-killer for the imperative approach.
>>
>> (The UA can tightly control battery and data usage in both the imperative
>> and declarative approaches described above, which is why I’m not focusing
>> on those here).
>>
>> We’re currently thinking of prototyping the imperative approach. But it
>> seems that the 2 main capabilities introduced incrementally in section B
>> (declarative push message actions, and the caches mechanisms) could provide
>> a viable plan B if the imperative approach turns out to use too much RAM..
>>
>> *Please don’t bikeshed the syntaxes yet*; at this stage the main open
>> questions are:
>>
>>
>>    1. Is this a reasonable vision for the set of imperative
>>    capabilities?
>>    2. Would such declarative capabilities be sufficient to address all
>>    important use cases?
>>    3. How easy would web developers find developing against such
>>    declarative APIs, compared to the imperative approach? The server would
>>    play a slightly greater role in driving the sync logic; but that may not be
>>    so terrible.
>>    4. How much RAM would such a declarative approach ultimately save? Is
>>    it worth it?
>>
>>
>> --John
>>
>>
>>
>
>
> --
> Matt Gaunt
> Chrome Developer Advocate
>



-- 
Matt Gaunt
Chrome Developer Advocate