Re: Sense of the WG on mediacapture-transform API

> On 18 Oct 2021, at 19:58, Bernard Aboba <Bernard.Aboba@microsoft.com> wrote:
> 
> At the WebRTC WG Virtual Interim meeting on Thursday, October 14, 2021 we discussed the path forward on mediacapture-transform API.  The slides from the meeting are available here:
> https://docs.google.com/presentation/d/1rdaRvl3h3viqvndyJVzwIAPtOpMjx_2u7q7MBHUwbyY/ <https://docs.google.com/presentation/d/1rdaRvl3h3viqvndyJVzwIAPtOpMjx_2u7q7MBHUwbyY/>
> 
> We would now like to get a "sense of the WG" on the following questions:
> 
> Question 1:  Should mediacapture-transform be based on streams? 

No.
At least not now.
This is a new pattern that we try to design, other APIs like WebCodecs encoder/decoder APIs have not adopted it due to issues like error handling.
Issues have been identified with adopting streams and we should make sure we have solutions to these.
I heard several times that there are solutions and would greatly welcome precise descriptions of these solutions.
The idea to adopt-and-let-it-fix-later is not appealing, especially since we are building something new, which is not consistent with other APIs (WebCodecs, WebAudio) in the field.

> 
> Question 2: Should mediacapture-transform support audio? 

No.
At least not now.
We have identified a clear technology gap with regards to video processing, let’s solve this gap.
TMK, we have not identified a clear technology gap with regards to audio processing, let’s identify the gap in terms of requirements and use cases before diving into API design.

> 
> Question 3: Should mediacapture-transform be exposed on the main thread? 

No.
At least not now.
We are trying to build a powerful API that should achieve the best possible performances, let’s do that and just that.
There are already existing APIs to do video processing in an easy but not efficient way.

That does not prevent us from adding main thread exposure later.
To do so, let’s list and discuss arguments in favour.
So far, I heard about platforms where doing processing in a worker would cause performance issues. Let’s identify these issues, measure them and investigate whether this is API or implementation related.
I also heard about ergonomics, let’s have clear examples of what ergonomics issues we are talking about.
I also heard about missing APIs that would be really useful to have in a worker. Let’s look at what is missing, in particular in the context of what we are trying to solve.


> 
>  
> Potential answers to the questions are "Yes", "No" or "Undecided". 
> 
> Please respond to this "Sense of the WG" poll by Monday, October 25, 2021 at midnight Pacific Time.