RE: WebRTC Next Version Use Cases: Now available!

Hello,
> There might be a potential additional requirement: hardware acceleration (say WebGL/WebGPU) should be available to process video content.

I started expanding this in Issue 17.<https://github.com/w3c/webrtc-nv-use-cases/issues/17>
Did you mean something else?
Kind Regards,
Aleksandar
________________________________
From: Bernard Aboba [Bernard.Aboba@microsoft.com]
Sent: Saturday, October 06, 2018 1:52 AM
To: youenn fablet
Cc: WebRTC WG
Subject: Re: WebRTC Next Version Use Cases: Now available!

Agree with your suggested next steps.  Sounds like you have
already begun to drill down on the requirements. Understanding
if they are real or just “nice to have” is very important.

Would you like to help with the discussion at TPAC and/or with the slides?

On Oct 5, 2018, at 2:15 PM, youenn fablet <yfablet@apple.com<mailto:yfablet@apple.com>> wrote:

Thanks Bernard,

This is a great document and I like the list of requirements.
I am curious about the next steps you would like to see happening there.

It seems to me we could do things like:
- Validate/extend the list of requirements
- List what is missing/sub-optimal in the web platform to fulfill the requirements.

Looking at section 3.3 funny hats for instance:
- Web Audio can be used to get access to the raw audio data and modify it (N20, N21, N22).
- Web Audio somehow supports N24 if we rewrite N24 to something like: "It must be possible to do efficient media manipulation out of the main thread."
- N23 supports seems to be missing but it is unclear whether this is a mandatory requirement that derives from the use cases.
The question that arises is whether Web Audio has any shortcoming with regards to WebRTC media processing.

If we look at video for the same section:
- Video element + Canvas + requestAnimationFrame allows supporting N20, N21, N22
- OffscreenCanvas supports N24
- N23 support seems to be missing
There might be a potential additional requirement: hardware acceleration (say WebGL/WebGPU) should be available to process video content.

Also, while N20, N21 and N22 are fulfilled, I believe the current web platform support has some shortcomings:
- requestAnimationFrame kicks in no matter whether the track has a new frame or not.
- Canvas is ARGB while track frames might be YUV which creates a conversion cost.
These shortcomings might limit the overall efficiency, thus limiting the processing in terms of frame rate and/or frame resolution.

Hope this helps,
Y

On Sep 4, 2018, at 11:44 AM, Bernard Aboba <Bernard.Aboba@microsoft.com<mailto:Bernard.Aboba@microsoft.com>> wrote:

Actually, a better link to view it is here: https://w3c.github.io/webrtc-nv-use-cases/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw3c.github.io%2Fwebrtc-nv-use-cases%2F&data=02%7C01%7CBernard.Aboba%40microsoft.com%7C7838b8800b32481833aa08d62b07b571%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636743709447322819&sdata=TgryuqswpHoqRpt93Y%2FMxqDVS3GSf873MCZXclk5tN8%3D&reserved=0>
________________________________
From: Bernard Aboba
Sent: Tuesday, September 4, 2018 11:39:00 AM
To: public-webrtc@w3.org<mailto:public-webrtc@w3.org>
Subject: WebRTC Next Version Use Cases: Now available!


An initial draft of one of the world's least remarkable documents has become available. By clicking on the links below, your eyes could be among the first to see it! Parental discretion advised!

To obtain your complementary copy of "WebRTC Next Version Use Cases", click here:

<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwebrtc-nv-use-cases&data=02%7C01%7CBernard.Aboba%40microsoft.com%7C7838b8800b32481833aa08d62b07b571%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636743709447322819&sdata=JXLMHd%2BxcVEsDms92XlubVutjBRLZ6GBeG%2FQsEaRgdc%3D&reserved=0>https://cdn.rawgit.com/w3c/webrtc-nv-use-cases/6643c1b4/index.html<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcdn.rawgit.com%2Fw3c%2Fwebrtc-nv-use-cases%2F6643c1b4%2Findex.html&data=02%7C01%7CBernard.Aboba%40microsoft.com%7C7838b8800b32481833aa08d62b07b571%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636743709447332882&sdata=xUtzcPVX1y3npF5NdWwSHA2fDifyjij9EyZo0b%2BJRXI%3D&reserved=0><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcdn.rawgit.com%2Fw3c%2Fwebrtc-nv-use-cases%2F6643c1b4%2Findex.html&data=02%7C01%7CBernard.Aboba%40microsoft.com%7C7838b8800b32481833aa08d62b07b571%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636743709447332882&sdata=xUtzcPVX1y3npF5NdWwSHA2fDifyjij9EyZo0b%2BJRXI%3D&reserved=0>


To enable readers to express their outrage and dismay, Issues can be filed here:

https://github.com/w3c/webrtc-nv-use-cases/issues<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fw3c%2Fwebrtc-nv-use-cases%2Fissues&data=02%7C01%7CBernard.Aboba%40microsoft.com%7C7838b8800b32481833aa08d62b07b571%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636743709447342834&sdata=rpXwVHt8t%2ByalKpNwCTv5MS87sEgyCs73xhi%2FJ5hDAs%3D&reserved=0>