[media-source] Dynamic addition/removal of SourceBuffers (#359)

dsilhavy has just created a new issue for https://github.com/w3c/media-source:

== Dynamic addition/removal of SourceBuffers ==
This issue is a follow up on the discussion we had in the W3C Media and Entertainment IG calls. See also `Additional Information` below.

# Overview
MSE based media players initialize the required `SourceBuffer` objects at the beginning of a media playback session. Typically, there will be one `SourceBuffer` for `audio` and one for `video`. After the initialization of the `SourceBuffers` they can not dynamically be removed and (re-)added again.

# Description / Background / Use Cases

## DASH multiperiod playback
In DASH based media streaming, the content is signaled in an MPD that can contain multiple periods. In a simple example, an MPD can consist of a main period, followed by an ad period followed by a main period again:

* Period 1: Main content with `audio` and `video` 
* Period 2: Ad content with `video` and **no** `audio`
* Period 3: Main content with `audio` and `video`.

In the above example, Period 2 does not contain an `AdaptationSet` of type `audio`. With the current implementation of the `MediaSource`, playback will stall as soon as one of the attached `SourceBuffers` runs dry and has no data. For Period 2 there is no audio data available, consequently the `SourceBuffer` of type `audio` can only be filled with "silent dummy data".
An alternative to feeding "dummy data" into the `SourceBuffer` is to fully reinitialize die `MediaSource` and the `SourceBuffers`. This will have an impact on the user experience and pre-buffered video data is lost.

## Media over QUIC
I am not actively following the work in MoQ but from my understanding one of the use cases might be related to this as well. In MoQ data can be prioritized, for instance it would be possible to assign a higher priority to the delivery of audio data than to the delivery of the video data in case of temporary network issues. 

To allow playback to continue in this example, the video `SourceBuffer` should temporarily be removed and readded again once there is sufficient network capacity to receive video data again.  

# Suggested Solution
To account for the scenarios described above and allow applications to continue with a smooth playback if one of the initially created `SourceBuffers` can not be filled with data, we suggest allowing the dynamic removal and (re-)addition of `SourceBuffers`.

# Additional Information
For related information, please refer to the meeting minutes of the W3C Media and Entertainment IG:

* [Joint meeting W3C/SVTA](https://www.w3.org/2024/12/10-me-minutes.html)
* [MEIG Monthly Meeting - 2025/01/07](https://www.w3.org/2025/01/07-me-minutes.html)

Please view or discuss this issue at https://github.com/w3c/media-source/issues/359 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 8 January 2025 14:15:45 UTC