- From: Chris Cunningham <chcunningham@google.com>
- Date: Fri, 25 Jun 2021 16:38:48 -0700
- To: public-media-wg@w3.org
Chrome supports exposing VideoEncoder, AudioEncoder, VideoDecoder, and AudioDecoder in Window contexts at this time. Our decision is based on the following arguments. Design ====== * These APIs were designed with asynchronous execution in mind [1]. They already function as a shim between the window or the worker thread and a pool of codec threads. No codec work occurs on the API thread beyond frame and chunk delivery. * [1] https://w3c.github.io/webcodecs/#control-thread-and-codec-thread * The UA’s “shim” is extensively tested by web platform tests [1] and uses native primitives. Forcing developers to find a library or implement their own saddles developers with unnecessary performance and maintenance burdens. * [1] https://github.com/web-platform-tests/wpt/tree/master/webcodecs * Life cycle (state) and back pressure (queue size) management attributes will skew across window and worker in ways that a developer will have to carefully consider to avoid excess resource usage [1, 2]. * [1] https://w3c.github.io/webcodecs/#videodecoder-interface * [2] https://w3c.github.io/webcodecs/#videoencoder-interface * Codec APIs are not new, they've existed for 20+ years on native platforms [1, 2, 3, 4]. There's extensive precedent for exposing these APIs without threading restrictions and there’s no precedent for limiting web APIs to workers. Such a radical departure from the status quo demands strong evidence (e.g., existing API limitations are for concrete security reasons). * [1] https://developer.apple.com/documentation/videotoolbox * [2] https://docs.microsoft.com/en-us/windows/win32/api/d3d11/nn-d3d11-id3d11videodecoder * [3] https://developer.android.com/reference/android/media/MediaCodec * [4] https://01.org/linuxmedia Developer preference ==================== * Of the participants in the Chrome origin trial, developers overwhelmingly voiced their preference for maintaining window exposure. The tally was 10 in favor of window exposure, 6 neutral, 1 ambivalent, 1 opposed [1]. * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741 * Many respondents found no performance reason to use workers and disliked the additional complexity that workers required [1]. * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741 * Some respondents found other APIs (discussed below) they wanted to use in conjunction with WebCodecs were not available in a worker [1]. * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741 * Many of the respondents continued to participate in the discussion on the issue [1]. We recommend reading through responses from AshleyScirra [2], bradisbell [3], jamespearce2006 [4], koush [5], and surma [6]. * [1] https://github.com/w3c/webcodecs/issues/211 * [2] https://github.com/w3c/webcodecs/issues/211#issuecomment-854919492 * [3] https://github.com/w3c/webcodecs/issues/211#issuecomment-833709528 * [4] https://github.com/w3c/webcodecs/issues/211#issuecomment-853652069 * [5] https://github.com/w3c/webcodecs/issues/211#issuecomment-854064846 * [6] https://github.com/w3c/webcodecs/issues/211#issuecomment-853706556 API availability ================ * Most UAs have not yet shipped OffscreenCanvas [1] or some even ImageBitmap [2], both of which are significant multi-year implementation efforts. * A worker only limitation means WebGL and WebGPU based applications must move their entire rendering path to a worker to avoid significant overhead from transferring VideoFrames back and forth between threads for texture upload. * This means more UAs will have to fully ship OffscreenCanvas along with WebCodecs, making it difficult to interoperate between UAs in a progressively enhanced fashion. I.e., we may see more Chrome-only applications with a worker limitation. * [1] https://caniuse.com/offscreencanvas * [2] https://caniuse.com/createimagebitmap * APIs which require user activation or a permission prompt [1, 2] are window bound, so even if most of their functionality is also available in a worker, initiation must start on the window. As such developers often just use the APIs in window. * [1] https://wicg.github.io/webusb/#enumeration * [2] https://wicg.github.io/serial/#serial-interface * Forthcoming APIs like WebGPU [1], with more intensive performance requirements than WebCodecs, are exposed in both window and worker scope to provide developers maximum flexibility. A worker restriction unnecessarily harms interoperability. * [1] https://www.w3.org/TR/webgpu/#navigator-gpu Performance =========== * Main thread contention may result in increased latency to perform codec control and process I/O. For use cases involving low latency and busy main threads (e.g. RTC), this concern motivates using WebCodecs in a worker. * For use cases that are either latency-flexible or have uncontended main threads, the above concerns are immaterial. Latency-flexible use cases include video editing, game editing, and audio workstations (see developer comments from jitter.video and scirra.com [1]). Uncontended use cases include apps that offer minimalist UX (see developer comments from vysor.io [2]). * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-848762741 * [2] https://github.com/w3c/webcodecs/issues/211#issuecomment-852308769 * On low-end devices, where the main thread is more easily contended, using workers cannot improve performance when the number of threads already exceeds the number of processor cores. * Window usage provides adequate throughput (>= 120 fps) and latency (>= 30fps) across a range of devices, even under significant thread contention. All tested cases could maintain 60fps with a 2-3 frame buffer. [1] * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-866424457 * Using a JavaScript shim to simulate window-exposure actually harms performance for some use cases. * In uncontended cases (apps with simple user experiences), the shim is slower than window on throughput by a ~2% margin. On mobile / low core count devices, window usage may have up to ~23% faster throughput than workers [1]. * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-866424457 * A shim results in ~2.67x the peak memory usage of a window only solution [1]. * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-868054131 * A shim results in ~20% slower time to the first frame in the best case, uncached, worker startup can add orders of magnitude more latency. Use cases decoding short audio or video segments may be overwhelmed by worker startup costs, resulting in user visible/audible latency. Examples include: Decoding a handful of video frames for an animated mouseover poster effect. Decoding a handful of samples from an audio track for an input driven AudioBufferSourceNode sound effect. * [1] https://github.com/w3c/webcodecs/issues/211#issuecomment-866424457 Given all of the above points, Chrome strongly recommends that VideoEncoder, AudioEncoder, VideoDecoder, and AudioDecoder be exposed in both windows and worker contexts.
Received on Friday, 25 June 2021 23:39:48 UTC