Re: [EXTERNAL] Call for Consensus (CfC) on Adoption of the Insertable Streams API specification

It seems to me a separate effort should happen for manipulation of raw frames after the decoder or before the encoder.
This seems mostly orthogonal to insertable streams given it should be applicable to both local and remote tracks.

One thing where we might want to link this raw frame manipulation API with WebRTC is when this API allows consuming/producing metadata, like head tracking.
In that case, we probably want encoders/decoders to be pass through of this metadata.

> On 3 Jun 2020, at 07:52, Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> wrote:
> 
> Thanks Bernard, sounds good. So a follow question could be: couldInsertable Streams somehow provide handles to GPU buffers which could enable WebGPU operations? Just thinking out loud (I should go and do some investigation myself).
>  
> From: Bernard Aboba <Bernard.Aboba@microsoft.com>
> Date: Tuesday, 2 June 2020 at 19:58
> To: Stefan LK <stefan.lk.hakansson@ericsson.com>
> Cc: "public-webrtc@w3.org" <public-webrtc@w3.org>
> Subject: Re: [EXTERNAL] Re: Call for Consensus (CfC) on Adoption of the Insertable Streams API specification
>  
> On Jun 2, 2020, at 2:00 AM, Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> wrote:
>>  
>> I’d like to see more work to determine if this model is also suitable for accessing raw media, and I’d also like to see overlap/sync vis a vis WebCodecs investigated.
>> Regarding N22: would it be worth looking into whether WebGPU (https://gpuweb.github.io/gpuweb/ <https://protect2.fireeye.com/v1/url?k=8f90bc3a-d1307e7c-8f90fca1-86fc6812c361-7251295517553986&q=1&e=daa517f5-5c14-48d2-a604-2f6cad31d9ac&u=https%3A%2F%2Fnam06.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fgpuweb.github.io%252Fgpuweb%252F%26data%3D02%257C01%257CBernard.Aboba%2540microsoft.com%257C8db5a4fef3a94a7679e408d806d32991%257C72f988bf86f141af91ab2d7cd011db47%257C0%257C0%257C637266852299530210%26sdata%3D%252BaA%252FeU2LsVhlpAjFBioVPdLOdsoygrQ3yzN8yo0ui6w%253D%26reserved%3D0>) could be useful?
>  
> [BA] Excellent points. Raw media is much larger than encoded, so performance is a concern. One thing under discussion in WebCodecs is whether it might be useful to provide handles to GPU buffers which could enable WebGPU operations on raw (or encoded) media.
>  
> Prototyping of WHATWG stream support in WebCodecs did not go well, and issues relating to WebRTC parity and DRM support have been raised. So at present Insertable Streams is much closer to a “bird in the hand” although we are not sure what songs this bird may be able to sing.