Re: [mediacapture-insertable-streams] Is using ReadableStream/WritableStream the right approach? (#4)

> Its only disadvantage would be running on the main thread by default
That is one disadvantage.

ScriptProcessorNode is an evidence of this issue for audio and the plan is to obsolete it, not complement it with Audio Worklet. Why should we reintroduce this potential issue?

> Experience in the field shows that the main thread can work fine for many use cases, and moving to workers is easy as well.
On certain devices, on certain configurations. The more we will see camera shooting with higher fps, the worse it might get.
I am not against main thread processing, but it seems this should be opt-in not the default.

There are other potential issues that might get solved (or be more acute) once the relationship between a track and its streams is made more precise. Currently the spec is light there so it is difficult to assess it:
- For instance, it is not clear how backpressure applies in general.
For audio this might be a no-go, loosing a frame will cause audio chirps. For camera, this might be beneficial in cases the processing pipe is too slow to process. That might help the case of a camera that is using a buffer pool and if each buffer in the buffer pool is in the pipe, capture will stop.
How do you anticipate MediaStreamTrackProcessor use the backpressure mechanism? Should MediaStreamTrackProcessor be able to define its backpressure mechanism in terms of number of frames that are queued?
- Similarly, how does MediaStreamTrack cloning refer to ReadableStream cloning? Is it guaranteed to get the same content between the two approaches?

Also, this somehow creates two APIs doing roughly the same thing: MediaStreamTrack and ReadableStream/WritableStream pair. API-wise, we would prefer sticking with MediaStreamTrack producers and consumers. Or do we anticipate other APIs to get raw content directly from streams and replace MediaStreamTrack progressively?
Ditto for MediaStreamTrackProcessor.writableControl which seems potentially somehow redundant with existing MediaStreamTrack APIs like enabled/disabled/applyConstraints. It would be good to clarify this, at least for me.

Talking of the signal control API, I am not clear about what happens if the processor does not process the signals or is not prepared to understand some signals (given it is extensible). I could think of some transforms just forgetting about sending the signals up in the chain.
So they could potentially pile in the stream, or would there be some backpressure as well to help the caller understand what is happening?
Or they could be lost in the middle of the chain. I am not sure how feasible it is, but for those kind of signals, it might be good to have a default behavior and allow the application to override it.

Maybe I should file separate issues but I want to ensure we pick the right design before we spend time on editing work.
Maybe we should do the exercise of comparing this model with other envisioned models like VideoTrackReader and the benefits will get clearer.

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/mediacapture-insertable-streams/issues/4#issuecomment-776538379 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 10 February 2021 08:37:45 UTC