Re: CfC: Should WebCodecs be exposed in Window environments?

Chrome supports exposing VideoEncoder, AudioEncoder, VideoDecoder, and
AudioDecoder in Window contexts at this time. Our decision is based on
the following arguments.

Design
======

* These APIs were designed with asynchronous execution in mind [1].
They already function as a shim between the window or the worker
thread and a pool of codec threads. No codec work occurs on the API
thread beyond frame and chunk delivery.
  * [1] https://w3c.github.io/webcodecs/#control-thread-and-codec-thread

* The UA’s “shim” is extensively tested by web platform tests [1] and
uses native primitives. Forcing developers to find a library or
implement their own saddles developers with unnecessary performance
and maintenance burdens.
  * [1] https://github.com/web-platform-tests/wpt/tree/master/webcodecs

* Life cycle (state) and back pressure (queue size) management
attributes will skew across window and worker in ways that a developer
will have to carefully consider to avoid excess resource usage [1, 2].
  * [1] https://w3c.github.io/webcodecs/#videodecoder-interface
  * [2] https://w3c.github.io/webcodecs/#videoencoder-interface

* Codec APIs are not new, they've existed for 20+ years on native
platforms [1, 2, 3, 4]. There's extensive precedent for exposing these
APIs without threading restrictions and there’s no precedent for
limiting web APIs to workers. Such a radical departure from the status
quo demands strong evidence (e.g., existing API limitations are for
concrete security reasons).

  * [1] https://developer.apple.com/documentation/videotoolbox
  * [2] https://docs.microsoft.com/en-us/windows/win32/api/d3d11/nn-d3d11-id3d11videodecoder
  * [3] https://developer.android.com/reference/android/media/MediaCodec
  * [4] https://01.org/linuxmedia

Developer preference
====================

* Of the participants in the Chrome origin trial, developers
overwhelmingly voiced their preference for maintaining window
exposure. The tally was 10 in favor of window exposure, 6 neutral, 1
ambivalent, 1 opposed [1].
  * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741

* Many respondents found no performance reason to use workers and
disliked the additional complexity that workers required [1].
  * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741

* Some respondents found other APIs (discussed below) they wanted to
use in conjunction with WebCodecs were not available in a worker [1].
  * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741

* Many of the respondents continued to participate in the discussion
on the issue [1]. We recommend reading through responses from
AshleyScirra [2], bradisbell [3], jamespearce2006 [4], koush [5], and
surma [6].
  * [1] https://github.com/w3c/webcodecs/issues/211
  * [2] https://github.com/w3c/webcodecs/issues/211#issuecomment-854919492
  * [3] https://github.com/w3c/webcodecs/issues/211#issuecomment-833709528
  * [4] https://github.com/w3c/webcodecs/issues/211#issuecomment-853652069
  * [5] https://github.com/w3c/webcodecs/issues/211#issuecomment-854064846
  * [6] https://github.com/w3c/webcodecs/issues/211#issuecomment-853706556

API availability
================

* Most UAs have not yet shipped OffscreenCanvas [1] or some even
ImageBitmap [2], both of which are significant multi-year
implementation efforts.
  * A worker only limitation means WebGL and WebGPU based applications
must move their entire rendering path to a worker to avoid significant
overhead from transferring VideoFrames back and forth between threads
for texture upload.
  * This means more UAs will have to fully ship OffscreenCanvas along
with WebCodecs, making it difficult to interoperate between UAs in a
progressively enhanced fashion. I.e., we may see more Chrome-only
applications with a worker limitation.
  * [1] https://caniuse.com/offscreencanvas
  * [2] https://caniuse.com/createimagebitmap

* APIs which require user activation or a permission prompt [1, 2] are
window bound, so even if most of their functionality is also available
in a worker, initiation must start on the window. As such developers
often just use the APIs in window.
  * [1] https://wicg.github.io/webusb/#enumeration
  * [2] https://wicg.github.io/serial/#serial-interface

* Forthcoming APIs like WebGPU [1], with more intensive performance
requirements than WebCodecs, are exposed in both window and worker
scope to provide developers maximum flexibility. A worker restriction
unnecessarily harms interoperability.
  * [1] https://www.w3.org/TR/webgpu/#navigator-gpu

Performance
===========

* Main thread contention may result in increased latency to perform
codec control and process I/O. For use cases involving low latency and
busy main threads (e.g. RTC), this concern motivates using WebCodecs
in a worker.

* For use cases that are either latency-flexible or have uncontended
main threads, the above concerns are immaterial. Latency-flexible use
cases include video editing, game editing, and audio workstations (see
developer comments from jitter.video and scirra.com [1]). Uncontended
use cases include apps that offer minimalist UX (see developer
comments from vysor.io [2]).
  * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741
  * [2] https://github.com/w3c/webcodecs/issues/211#issuecomment-852308769

* On low-end devices, where the main thread is more easily contended,
using workers cannot improve performance when the number of threads
already exceeds the number of processor cores.

* Window usage provides adequate throughput (>= 120 fps) and latency
(>= 30fps) across a range of devices, even under significant thread
contention. All tested cases could maintain 60fps with a 2-3 frame
buffer. [1]
  * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-866424457

* Using a JavaScript shim to simulate window-exposure actually harms
performance for some use cases.
  * In uncontended cases (apps with simple user experiences), the shim
is slower than window on throughput by a ~2% margin. On mobile / low
core count devices, window usage may have up to ~23% faster throughput
than workers [1].
    * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-866424457
  * A shim results in ~2.67x the peak memory usage of a window only
solution [1].
    * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-868054131
  * A shim results in ~20% slower time to the first frame in the best
case, uncached, worker startup can add orders of magnitude more
latency. Use cases decoding short audio or video segments may be
overwhelmed by worker startup costs, resulting in user visible/audible
latency. Examples include: Decoding a handful of video frames for an
animated mouseover poster effect. Decoding a handful of samples from
an audio track for an input driven AudioBufferSourceNode sound effect.
    * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-866424457


Given all of the above points, Chrome strongly recommends that
VideoEncoder, AudioEncoder, VideoDecoder, and AudioDecoder be exposed
in both windows and worker contexts.

Received on Friday, 25 June 2021 23:39:48 UTC