Re: [mediacapture-region] What makes CropTarget special to require an asynchronous creation? (#17)

I would first like to get consensus on one API design point. Hopefully, we can all agree on something like:
1. For any new API, we try to make it synchronous if we can.
2. If the algorithm requires some hopping to other threads/processes/environments, we switch to async.
3. If synchronous implementations are overly complex, we switch to async.

AIUI, Chrome current implementation is asynchronous, and @eladalon1983 is stating that doing a synchronous implementation is overly complex.
@jan-ivar and @youennf think that such a synchronous implementation is not complex. Some implementation strategies and already implemented APIs that deal with the same issue have been suggested.
The rest of the message (sorry for its length) is describing that.

> * **MessageChannel:**
>   
>   * If implementing with direct communication between the processes, the risk involved is a necessary evil. This cannot be said for CropTarget. Discoverability in either direction is not a requirement here, and confers little to no benefit.
>   * If implementing with mediation via another process, the story gets more complicated. A valid implementation can hide that it's asynchronically minting identifiers, behind the moment of posting the MessageChannel to the other document. (Some compromises are required, though.) But I don't want to discuss this because it would lose track of the topic - see below.

@eladalon1983, I think we do agree MessagePort can be created and transferred cross process synchronously.
Can you validate Chrome implementation of MessagePort creation/transfer, in particular whether Chrome is minting asynchronously identifiers for MessagePorts?

FWIW, given we now have such MessagePort, we can reuse MessagePort to implement CropTarget.
Below algorithm is taking the hypothesis that CropTarget is transferable and can be used only once (these limitations can be easily fixed by recreating MessageChannels as required):
1. At CropTarget creation, create a MessageChannel, set port1 to a slot of the element, and port2 to a CropTarget slot.
2. When transferring CropTarget to another environment, transfer CropTarget.port2, and recreate a new CropTarget with the transferred port2
3. When calling cropTo with the new CropTarget, transfer CropTarget.port2 to the process doing the capture.
4. In the capture process, use port2 to communicate with the element (through port1) to gather the necessary states to start the actual cropping. 

Steps 1 and 2 can be done synchronously.
Steps 3 and 4 are asynchronous which is fine since they are run as part of cropTo which returns a promise.

> * **MediaStreamTrack:**
>   
>   * These are **not** currently transferrable in Chrome - not synchronously, not at all. To claim it's possible one needs to present an implementation  as proof, not a specification. (Does it work on Safari or Firefox yet? [This demo](https://cheerful-zany-editorial.glitch.me/) suggests that it's not, as of 2022-03-31.)
>   * My colleauges working on mst-transferability tell me that they are running into **a lot** of issues **precisely** because of the requirement that they be "synchronously transferable".

It would be nice to hear about the exact MST issues.
I would bet this is due to MST complex state handling.
CropTarget has no changing state, which makes it a much simpler object to transfer.
That said, video MST are probably already transferable synchronously in Chrome by doing the following:
1. Create a video MediaStreamTrack from canvas
2. Get the ReadableStream from MediaStreamTrack using MediaStreamTrackProcessor
3. Transfer the ReadableStream
4. Recreate the MediaStreamTrack from the transferred ReadableStream using VideoTrackGenerator

Again step 1, 2 and 3 are all synchronous, from creation of a MediaStreamTrack to transferring the MediaStreamTrack via ReadableStream.

> * **RTCDataChannel:**
>   
>   * As of the time of this writing, these don't appear to be transferable in neither [the specification](https://w3c.github.io/webrtc-pc/#webidl-1143016005) nor [Chrome's implementation](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/peerconnection/rtc_data_channel.idl).

It is specified in https://w3c.github.io/webrtc-extensions/#rtcdatachannel-extensions.
It is implemented in Safari by reusing pre-existing mechanisms.
One reason why it might not be as difficult to implement as MST is that we restrict when RTCDataChannel can be transferred. This simplifies a lot the state handlings.
CropTarget is much simpler to transfer than RTCDataChannel since it has no changing state.

The fourth implementation that was brought to the discussion is the following:
1. CropTarget stores an identifier of the process where lives the element (or the document environment ID which is generated for each document before hand, so can be retrieved synchronously).
2. CropTarget stores a locally generated identifier of the element.
3. Element is identified by the pair of these two identifiers
4. CropTarget is serialised by serializing these two IDs.
5. When calling cropTo, the capture process identifies the element process through the first ID and gathers element information through the second ID.

Steps 1 to 4 can be done synchronously.

-- 
GitHub Notification of comment by youennf
Please view or discuss this issue at https://github.com/w3c/mediacapture-region/issues/17#issuecomment-1127398984 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 16 May 2022 08:46:28 UTC