[mediacapture-transform] A review (#38)

jan-ivar has just created a new issue for https://github.com/w3c/mediacapture-transform:

== A review ==
This is a review in response to a [Call for Review](https://lists.w3.org/Archives/Public/public-webrtc/2021May/0002.html) expiring today. I was going to open issues as requested, only to find they mostly exist already — many predating the CFR even — but may be hard to find. So I'm writing this overview instead.

Outstanding design issues I see as blockers:
 1. Don’t expose raw realtime media on main thread, since there's a precedent that says this is bad https://github.com/w3c/mediacapture-transform/issues/23#issuecomment-842585817
 2. Limit to solving video for now, or explain why AudioWorklet is insufficient, to not reinvent it https://github.com/w3c/mediacapture-transform/issues/29
 3. Lack of a threading model https://github.com/w3c/mediacapture-transform/issues/27
 4. Lack of algorithms https://github.com/w3c/mediacapture-transform/issues/36
 5. Controlling channel mechanism is unclear https://github.com/w3c/mediacapture-transform/issues/24
 4. Unclear memory I/O design given that a VideoFrame may be in GPU memory https://github.com/w3c/mediacapture-transform/issues/34 (Funny hat: GPU→CPU→GPU?)
 5. Can one JS document hose a browser's camera capturing ability? https://github.com/w3c/mediacapture-transform/issues/37

The other issue is that there seem to be massive missing areas in WebCodecs that we rely on:
 1. VideoFrame seems optimized for WebCodecs, not funny hats, re copies — Is there room to question immutable VideoFrames vs e.g. lockable mutable ones when frames are *not* in GPU?
 2. Pixel formats seem underspecified, https://github.com/w3c/webcodecs/issues/165 and https://github.com/w3c/webcodecs/issues/200 vs. e.g. earlier [discarded ideas](https://w3c.github.io/mediacapture-worker/#pixellayout).

There also appears to be a lack of documentation or working samples to the point where any web developer can use it.

This makes it hard to see the big picture or validate whether the overall design addresses the listed use cases (a list that judging by the examples given, does not appear to require interfacing with WebCodecs — is that an omission?)

Please view or discuss this issue at https://github.com/w3c/mediacapture-transform/issues/38 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Tuesday, 18 May 2021 02:02:27 UTC