[media-source] [DRAFT] ManagedMediaSource: Explainer (#320)

jyavenard has just created a new issue for https://github.com/w3c/media-source:

== [DRAFT] ManagedMediaSource: Explainer ==
### Definitions

A “managed” MediaSource is one where more control over the MediaSource and its associated objects has been given over to the User Agent.

### Introduction

The explicit goal of the Media Source Extensions specification is to transfer more control over the streaming of media data from the User Agent to the application running in the page. This transfer of control can and has added points of inefficiencies, where the page does not have the same level of capabilities, knowledge, or even goals as the User Agent.
Examples of these inefficiencies include the management of buffer levels, the timing and amount of network access, and media variant selection. These inefficiencies have largely been immaterial on relatively powerful devices like modern general purpose computers. However, on devices with narrower capabilities, it can be difficult to achieve the same quality of playback with the MediaSource API as is possible with native playback paths provided by the User Agent.

The goal of the ManagedMediaSource API is to transfer some control back to the User Agent from the application for the purpose of increasing playback efficiency and performance, while retaining the ability for pages to control streaming of media data.

### Goals

* Make it easier for media web site authors to support streaming media playback on constrained capability devices.
* Allow User Agents to react to changes in available memory and networking capabilities.
* Reduce the power impact of MediaSource APIs.

### Non-Goals



### Scenario

#### Low-memory availability

A user loads a MSE-based website on a device with a limited amount of physical memory, and no ability to swap. The user plays some content on that website, and pauses that content. The system subsequently faces memory pressure, and requires applications (including the User Agent) to purge unused memory. If not enough memory is made available, those applications (including the User Agent) may be killed by the system in order to free up enough memory to perform the operation triggering this memory pressure.

The User Agent runs a version of the “Coded Frame Eviction” algorithm, removing ranges of buffered data in order to free memory for use by the system. At the end of this algorithm, the User Agent fires a “bufferedchange” event at every SourceBuffer affected by this algorithm, allowing the web application to be notified that it may need to re-request purged media data from the server before beginning playback.

#### Memory availability notification

When a call to appendBuffer is rejected with a QuotaExceededError exception, it can  indicate the amount of excessive bytes or time that caused the error.

#### Network Streaming changes

Currently, a web application is allowed to append media data into a Source Buffer at any time, up until that Source Buffer’s “buffer full flag” is set, indicating no additional data is allowed to be appended. However, a constrained device may want to coalesce network use into a small window, and allow the network to query for battery and bandwidth reasons. 

The User Agent would fire a “startstreaming” event at the MediaSource, indicating that the web application should begin streaming new media data. It would be up to the User Agent to determine when streaming should start, and could take current buffer levels, current time, network conditions, and other networking activity on the system.

When the User Agent determines that no further media streaming should take place, it would fire a “stopstreaming” event at the MediaSource, indicating to the web application that enough media data had been buffered to allow playback to continue successfully.

Optionally, the User Agent may enforce these streaming events by rejecting calls to appendBuffer() by returning a “NotAllowedException”.


#### Media variant selection

Currently, the player has few tools to determine the possible quality level of the media being played. From measuring the speed at which a segment download was performed, assessing the available dimensions available to render the video or how many frames got dropped when played.
The User Agent can access further details beforehand: from the speed of the networking medium, to the user's choice of preferring low data usage.
The User Agent may suggest a preferred streaming quality.


### WebIDL
```
enum PreferredQuality {
    "low",
    "medium",
    "high"
}

[Exposed=(Window,DedicatedWorker)]
interface ManagedMediaSource : MediaSource {
    constructor();

    readonly   attribute PreferredQuality quality;
               attribute EventHandler     onqualitychanged;

               attribute EventHandler     onstartstreaming;
               attribute EventHandler     onendstreaming;
                   
                  static boolean          isTypeSupported (DOMString type);                   
};

[Exposed=Window,DedicatedWorker]
interface BufferedChangeEvent : Event {
    constructor(DOMString type, optional BufferedChangeEventInit eventInitDict);

    [SameObject] readonly attribute TimeRanges? addedRanges;
    [SameObject] readonly attribute TimeRanges? removedRanges;
};

dictionary BufferedChangeEventInit : EventInit {
    TimeRanges? addedRanges = null;
    TimeRanges? removedRanges = null;
};

[Exposed=(Window,DedicatedWorker)]
interface ManagedSourceBuffer : SourceBuffer {
               attribute EventHandler     onbufferedchange;
};
```

### Usage example 

```
async function setUpVideoStream(){
  // Specific video format and codec
  const mediaType = 'video/mp4; codecs="mp4a.40.2,avc1.4d4015"';
  
  // Check if the type of video format / codect is supported.
  if (!ManagedMediaSource.isTypeSupported(mediaType)) {
    return; // Not supported, do something else.
  }

  // Set up video and its managed source.
  const video = document.createElement("video");
  const source = new ManagedMediaSource();
  
  video.disableRemotePlayback = true;
  video.controls = true;
  
  await new Promise((resolve) => {
    video.src = URL.createObjectURL(source);
    source.addEventListener("sourceopen", resolve, { once: true });
    document.body.appendChild(video);
  });
  
  const sourceBuffer = source.addSourceBuffer(mediaType);
  
  // Set up the event handlers
  sourceBuffer.onbufferedchange = (e) => {
    console.log("onbufferedchange event fired.");
    console.log(`Added Ranges: [${timeRangesToString(e.addedRanges)}]`);
    console.log(`Removed Ranges: [${timeRangesToString(e.removedRanges)}]`);
  };
  
  source.onstartstreaming = async () => {
    const response = await fetch("./videos/bipbop.mp4");
    const buffer = await response.arrayBuffer();
    await new Promise((resolve) => {
      sourceBuffer.addEventListener("updateend", resolve, { once: true });
      sourceBuffer.appendBuffer(buffer);
    });
  };
  
  source.onendstreaming = async () => {
    // Stop fetching new segments here
  };
}

// Helper function...
function timeRangesToString(timeRanges) {
  const ranges = [];
  for (let i = 0; i < timeRanges?.length; i++) {
    const range = [timeRanges.start(i), timeRanges.end(i)];
    ranges.push(range);
  }
  return ranges.toString();
}

<body onload="setUpVideoStream()"></body>
```

### Privacy considerations

TODO: discuss potential privacy protections if multiple origins try poke at this at the same time. 
A concern is providing visibility to preferred quality if it is based on networking condition such as cellular or wifi etc.

### Other

To consider from MSE v2:

* Define behaviour for playback over unbuffered range.


Please view or discuss this issue at https://github.com/w3c/media-source/issues/320 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Monday, 29 May 2023 08:48:25 UTC