[w3ctag/design-reviews] WebCodecs support for AV1 screen content coding tools (Issue #912)

Moin moin TAG!

I'm requesting a TAG review of a WebCodecs extension to add support for AV1 screen content coding tools ([asked here](https://groups.google.com/a/chromium.org/g/blink-dev/c/BLAW7YO17jE/m/JTlF3HirBgAJ))

This adds AV1EncoderConfig (a dictionary containing a boolean `forceScreenContentTools` (a term from the AV1 bitstream spec)) to the VideoEncoderConfig along these lines: 
```
encoder.configure({
  codec: 'av01.0.04M.08',
  width: 1920, height: 1080, bitrate: 2_000_000, framerate: 5, 
  av1: { forceScreenContentTools: true},
}); 
```
This allows an application to encode “screen content”, in particular presentation slides, in a more efficient way supported by the AV1 codec. This material is typically static, often includes text, a limited set of colors, lots of repetitive content (e.g. straight lines, shapes) for which the encoder can optimize.

See the explainer for a lot of visual examples. This AV1 feature is already supported by WebRTC and enabled for screen sharing MediaStreamTracks so this increases platform consistency.

  - Explainer: [here, with lots of pictures, some of them blurry](https://github.com/fippo/webrtc-explainers/blob/gh-pages/av1-screencontent/index.md)
  - Specification URL: [VideoEncoderOptions](https://w3c.github.io/webcodecs/#video-encoder-options) which has [codec-specific extensions](https://w3c.github.io/webcodecs/codec_registry.html#video-codec-registry), here for [AV1](https://www.w3.org/TR/webcodecs-av1-codec-registration/)
  - Tests: n/a
  - Security and Privacy self-review²: extension of [WebCodecs](https://github.com/WICG/web-codecs/blob/master/security-privacy-questionnaire.md), wiring up an encoder feature should not change this substantially
  - Primary contacts (and their relationship to the specification):
      - Philipp Hancke (@fippo), Microsoft
      - Bernard Aboba (@aboba), Microsoft, WebCodecs spec editor
  - Organization(s)/project(s) driving the specification: Chromium
  - Key pieces of existing multi-stakeholder review or discussion of this specification: [spec pr](https://github.com/w3c/webcodecs/pull/662), [Media WG minutes](https://lists.w3.org/Archives/Public/public-media-wg/2023Apr/0007.html)
  - External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
https://chromestatus.com/feature/6307770441138176
Further details:

  - [x] I have reviewed the TAG's [Web Platform Design Principles](https://www.w3.org/TR/design-principles/)
  - Relevant time constraints or deadlines: [ideally Chrome M121 but not urgent](https://chromiumdash.appspot.com/schedule)
  - The group where the work on this specification is currently being done: Media Working Group

You should also know that...

How extensibility is handled is probably the more interesting thing to review!

We'd prefer the TAG provide feedback as (please delete all but the desired option):
  💬 leave review feedback as a **comment in this issue** and @-notify @fippo



-- 
Reply to this email directly or view it on GitHub:
https://github.com/w3ctag/design-reviews/issues/912
You are receiving this because you are subscribed to this thread.

Message ID: <w3ctag/design-reviews/issues/912@github.com>