Re: Draft notes from today's meeting

Since the google.com domain doesn't permit sharing to "everyone", here's 
a copy of the minutes.


On 6/22/21 10:25 PM, Bernard Aboba wrote:
> Thank you to Guido for taking notes at today's meeting:
> https://docs.google.com/document/d/1y2LEGdGxZqpKfACjVlHzaiI5e38DZRM0T8Odnftcl0I/ 
> <https://docs.google.com/document/d/1y2LEGdGxZqpKfACjVlHzaiI5e38DZRM0T8Odnftcl0I/>
>
>
> Please send comments/additions/deletions to me.


**

*Notes from WebRTC Meeting. 2021-06-22.*

*

Dan Sanders (Google)presents why WebCodecs decided not to use streams 
and why VideoTrackReader was removed in favor of mediacapture-transform.

  *

    The streams control/states is not a good fit for codecs state machines.

  *

    Required knowledge about lots of minor details of streams


Youenn:

Any other benefits or downsides apart from optimized transferring?

A: Clean model for transferring. Buffering.

More boilerplate not seen as a blocker (Youenn's wording).


Harald:

Question about transferring chunks?

A: Chunks not transferable. Issue handled at the C++ layer.


Jan-Ivar:

Fetch, WebTransports as other examples of streams. Is it possible to 
polyfill streams on top of WebCodecs?

A: WebCodecs uses streams for ImageDecoder, good integration with Fetch.

A: There are use cases where streams can work well for WebCodecs, where 
a subset of the functionality is used. A Streams wrapper would work well 
in these cases.

Examples that are not good: seeking state transitions are not a good 
match for streams.

Key frames OK with streams.


Tim:

Two things trying to consume the same stream. Is clone a problem?

A: Implementation does not clone. Tee() can lead to surprises.


Harald reports he wrote an adapter.


Bernard:

Q: How to handle congestion?

A (Harald): The demo doesn't handle congestion.


END OF SEGMENT


Harald (Google): Streams or No Streams?


Needs: Efficient media processing, minimum disruption of existing APIs, 
processing off-main thread.

For video, we already had canvas. It worked, but it's a bad idea.

Audio and video are inherently sequenced. In WebRTC timing is carried in 
stamps, not system clock. No need to sync with system clock except to 
play out at the same time they arrive.


The MediaStreamTrack API is strictly a control platform. It does not 
directly interact with the media, but directs how to treat it.


Streams provide good properties.

Not tied to the system clock.

Existing feature.

Tee is a footgun, but a known one.


Non-media events: we don't have a good solution at the moment.

Proposal: Don't solve the problem in the first iteration.


Proposal: Generate stream from track and generate track from stream. 
Stop there, get feedback from the field and then move to control. Decide 
that building on streams is a good idea.


END OF SEGMENT

Youenn (Apple):  VIEW OF STREAMS

Backpressure: Mostly unused by existing sources (e.g., microphones, 
cameras, canvas, PC). Maybe some future sources would use it. 
Backpressure might be useful.


Buffering: Streams are good at buffering. Potential footgun for video 
processing. For example, camera capturers that have a buffer pool. Hard 
to know how much is buffered in complex cases with multiple streams and 
transforms.

It should be left to the application, or be very visible.

Callback-based would not do buffering.


Locking: People may not be familiar with streams. Locking is a mismatch 
with tracks (according to the presenter) which can have multiple sinks.

Tee does not match track cloning. It's a footgun.

There's boilerplate.


Transferability: Streams are transferable.  There is rough consensus on 
making tracks transferable.  It's better to transfer the track than the 
stream.

Stream transferability is nowhere, according to the presenter.


Piping:


Proposal: Make a full comparison and wait for that comparison to make a 
proposal.


Q: Jan-Ivar: Transferring tracks to workers should be better. He agrees 
with Youennf on this.

Tee is evolving. You can have streams without buffering.  In streams by 
default there is no buffering.

Streams is a good pattern for some things, like promises for backpressure.

MSTP is a platform object, so there is no inherent reason why buffering 
should be a problem.


Jan-Ivar claims that it's pretty obvious if streams are buffering or 
not. High watermark allows you to control that.


Harald: transferring a control surface shouldn't be necessary to 
transfer media for processing.

Buffering is an issue.


Adam Rice:

Streams provide ergonomic benefits. You can write more than you can read 
without losing data, but in some cases such as tracks and WebTransport 
you want to drop old data and keep fresh data. WebTransport uses a 
buffer separate from streams with max age, you always get fresh data.

If you design APIs so that patterns that make sense you avoid the footguns.


A: Wasn't aware of buffering zero, but will look into it. (Note by 
notetaker: it's what MSTP does)


Jan-Ivar:

Streams provide a pattern for ensuring that only one thing is processed 
at a time.

Plain callbacks if they don't have promises they may explode the stack. 
Backpressure is good to avoid this.

A: I agree with backpressure being useful. The proposal will deal with that.



END OF SEGMENT


There is a poll to get a sense of the room, not a vote, so not a signal 
for consensus, and it might be skewed.


Decision: Not call for a decision.

4 in favor of streams, 5 in favor of callbacks, 4 in favor of need more 
information.



END OF SEGMENT


Jan-Ivar (Mozilla): Safer integrated Web presentations.

Reuse site isolation/opt-in header for getViewportMedia in 
getDisplayMedia for preferential treatment. Freeze if there is navigation.


Isolated browser as a new displaySurface. Put warnings when not isolated.


Elad: Proposes to rephrase such that "preferential treatment" is not

A: Agree.

Elad: Freeze might be bad for the user.

  A: The idea is to guarantee that the guarantee is kept.


Youenn: Echo's Elad's idea. Note: Couldn't follow the rest.

A: Rhetorical device for introducing the concept.


Harald: Breaking compatibility with existing behavior based on target 
page properties is bad, bad, bad. If capture asks for safety it's OK to 
modify behavior. If capture doesn't have opinions on safety, it 
shouldn't . Looks like a good idea if a new 
option/method/flag/constraint to signal that you want safety.


Henrik Böstrom: This sounds like something the browser should decide.


Elad: I've proposed a pauseOnNavigation constraint.


A: There have been lots of proposals to try to solve this problem.

The idea is to move in the direction of favoring safer capture of pages 
that have opted-in in being captured.


Harald: There is interest, but without breaking existing behavior.

A: There is no existing behavior.


END OF SEGMENT


Screen Capture slide controls

Encourage the WG to standardize a baseline for web presentation controls.

Proposal: Slides controller that allows defining presentation controls, 
and some events to communicate UI events for the controls.

API is secure context, throttled, and requires site isolation.


Youenn: What to do to answer these questions. How do you suggest we make 
progress?

A: Submit to the WG for consideration.


Harald: It's out of charter. There's another WG working on this. We have 
no obligation to do something outside our charter. So, let's not do it. 
Anything outside the capture is out of charter.

A: This is in the charter. Web presentations.


Elad: Sounds like an interesting solution. Looks very narrow, just 
next/prev slide, but this is not an argument against capture handle.

A:


END OF SEGMENT


Jan-Ivar (Mozilla): Capture of an iframe.

Proposal: Only allow capture of self iframe and have a frame to transfer 
its track to another place. The only options are SELF, TAB.


Elad: Also allow NODE/FRAME to capture a specific iframe in the tab.


Youenn: Why do you not like NODE/FRAME? and what about the default?

A: Frame ID, puts us back to IDs. Tracks are transferable.


Youenn: Don't see a big difference between iframe and just SELF 
implementation wise.

Don't see an issue with capturing a frame.

A: Not a good answer at the moment.


END OF SEGMENT


Elad (Google): Conditional focus.

Let the application decide if focus should switch to a captured tab.

Time-limited focus() API to signal focusing to the captured tab or not.


Tim: Messing with focus without the user being involved is a bad idea.

A: We are already doing it.

Tim: It should be a decision of the user (e.g., in the picker).


Youenn: I share Tim's concerns. Once the track is available it's too late.

Maybe provide some hints. Or have a way to influence the focus decision.


Jan-Ivar: There are some security concerns. Flashbacks to web pages that 
open browsers.

Don't think the application knows best what to do.

The use case is presentation, but the tool is a blowtorch.

A: Not clear how the UA knows, or how to produce a sensible UI where a 
checkbox changes depending on what is selected.


Harald: I think this solves a real problem.


Proposal: Continue discussion on github.


END OF SEGMENT


Summary: Bernard (Microsoft)


Callbacks vs Stream.

Decided to have a concrete presentation for the callback API in the next 
meeting.

It would be good to have it before the meeting.


Harald: Maybe it's not either or.

Youenn: Maybe both can be merged.

Tim: neither of them are perfect. Maybe we'll end up with some compromise.

Jan-Ivar: Other things need to be solved apart from the surface. For 
example, main-thread exposure.  We need to continue discussing it.




*