[mediacapture-record] Clarification needed for mime type handling (#170)

Pehrsons has just created a new issue for https://github.com/w3c/mediacapture-record:

== Clarification needed for mime type handling ==
Let's start by looking at the `MediaRecorderOptions` `mimeType` attribute, where the spec says:
> The container and codec format(s) [RFC2046] for the recording, which may include any parameters that are defined for the format. If the UA does not support the format or any of the parameters specified, it MUST throw a `NotSupportedError` `DOMException`. If this paramater is not specified, the UA will use a platform-specific default format. The container format, whether passed in to the constructor or defaulted, will be used as the value of the `mimeType` attribute.

This makes it sound like the `MediaRecorder`, upon running the constructor, will lock its `mimeType` attribute to either the passed in value, or a defaulted value (if invalid it throws, so all these are valid and supported).

Then if we look at the `MediaRecorder` `mimeType` attribute, the spec says:
> The MIME type [RFC2046] that has been selected as the container for recording. This entry includes all the parameters to the base mimeType. The UA should be able to play back any of the MIME types it supports for recording. For example, it should be able to display a video recording in the HTML <video> tag. The default value for this property is platform-specific. 

This also sounds like it's read-only and never changes, but it seems to go against the language for the `MediaRecorderOptions` `mimeType` attribute in that it says "This entry includes all the parameters to the base mimeType". I take it "all the parameters" (definition?) are meant to be added to whatever was passed in with `MediaRecorderOptions`?
The contradiction lies in that the language for `MediaRecorderOptions.mimeType` says "The container format, whether passed in to the constructor or defaulted, will be used as the value of the `mimeType` attribute".

Clarifying this would be Part 1 of this issue.

Part 2 is minor, but "all the parameters" in the paragraph above would be good to clarify.
- Is it a MUST to define all codecs to be used before tracks are known? Are there other parameters that MUST be listed explicitly? "all" is very broad.

Part 3 is to clarify what we are allowed to encode once the `mimeType` has been locked onto.
- Can an audio mime type (e.g., `audio/webm;codecs=opus`) encode video?
- Can a video mime type (e.g., `video/webm;codecs=vp8`) encode audio?
- Can an audio/video mime type (e.g., `video/webm;codecs=opus,vp8`) encode only audio?
- Can an audio mime type declare a video codec and encode audio and video?

To gather some information on existing behavior I wrote a fiddle to test this, https://jsfiddle.net/pehrsons/h4rwbn1e/.
It gives per the below,

Legend:
- `ctor-mime` is the recorder's `mimeType` attribute just after running the MediaRecorder constructor
- `recording-mime` is the recorder's `mimeType` attribute just after the first `dataavailable` event (timeslice is provided)
- `blob-mime` is the `type` attribute of the `Blob` in the first `dataavailable` event
- `recorded tracks` is a list of the tracks (and their kinds) that were the result of decoding the recording, i.e., to see if the `mimeType` attribute was lying

Firefox (though note that this is the behavior I'm intending to make spec compliant):
```
[MediaRecorder "" [audio]]: ctor-mime: ""; recording-mime: "audio/ogg"; blob-mime: "audio/ogg"; recorded tracks: [audio]
[MediaRecorder "" [video]]: ctor-mime: ""; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [video]
[MediaRecorder "" [audio,video]]: ctor-mime: ""; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [audio,video]
[MediaRecorder "audio/webm;codecs=opus" [audio]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "audio/ogg"; blob-mime: "audio/ogg"; recorded tracks: [audio]
[MediaRecorder "audio/webm;codecs=opus" [video]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [video]
[MediaRecorder "audio/webm;codecs=opus" [audio,video]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [audio,video]
[MediaRecorder "video/webm;codecs=vp8" [audio]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "audio/ogg"; blob-mime: "audio/ogg"; recorded tracks: [audio]
[MediaRecorder "video/webm;codecs=vp8" [video]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [video]
[MediaRecorder "video/webm;codecs=vp8" [audio,video]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [audio,video]
[MediaRecorder "video/webm;codecs=opus,vp8" [audio]]: ctor-mime: "video/webm;codecs=opus,vp8"; recording-mime: "audio/ogg"; blob-mime: "audio/ogg"; recorded tracks: [audio]
[MediaRecorder "video/webm;codecs=opus,vp8" [video]]: ctor-mime: "video/webm;codecs=opus,vp8"; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [video]
[MediaRecorder "video/webm;codecs=opus,vp8" [audio,video]]: ctor-mime: "video/webm;codecs=opus,vp8"; recording-mime: "video/webm"; blob-mime: "video/webm"; recorded tracks: [audio,video]
[MediaRecorder "audio/webm;codecs=opus,vp8" [audio]]: Exception; NotSupportedError: Operation is not supported
[MediaRecorder "audio/webm;codecs=opus,vp8" [video]]: Exception; NotSupportedError: Operation is not supported
[MediaRecorder "audio/webm;codecs=opus,vp8" [audio,video]]: Exception; NotSupportedError: Operation is not supported
```

Chrome:
```
[MediaRecorder "" [audio]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "audio/webm;codecs=opus"; blob-mime: "audio/webm;codecs=opus"; recorded tracks: [audio]
[MediaRecorder "" [video]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "video/webm;codecs=vp8"; blob-mime: "video/webm;codecs=vp8"; recorded tracks: [video]
[MediaRecorder "" [audio,video]]: ctor-mime: "video/webm;codecs=vp8,opus"; recording-mime: "video/webm;codecs=vp8,opus"; blob-mime: "video/webm;codecs=vp8,opus"; recorded tracks: [audio,video]
[MediaRecorder "audio/webm;codecs=opus" [audio]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "audio/webm;codecs=opus"; blob-mime: "audio/webm;codecs=opus"; recorded tracks: [audio]
[MediaRecorder "audio/webm;codecs=opus" [video]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "audio/webm;codecs=opus"; blob-mime: "audio/webm;codecs=opus"; recorded tracks: [video]
[MediaRecorder "audio/webm;codecs=opus" [audio,video]]: ctor-mime: "audio/webm;codecs=opus"; recording-mime: "audio/webm;codecs=opus"; blob-mime: "audio/webm;codecs=opus"; recorded tracks: [audio,video]
[MediaRecorder "video/webm;codecs=vp8" [audio]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "video/webm;codecs=vp8"; blob-mime: "video/webm;codecs=vp8"; recorded tracks: [audio]
[MediaRecorder "video/webm;codecs=vp8" [video]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "video/webm;codecs=vp8"; blob-mime: "video/webm;codecs=vp8"; recorded tracks: [video]
[MediaRecorder "video/webm;codecs=vp8" [audio,video]]: ctor-mime: "video/webm;codecs=vp8"; recording-mime: "video/webm;codecs=vp8"; blob-mime: "video/webm;codecs=vp8"; recorded tracks: [audio,video]
[MediaRecorder "video/webm;codecs=opus,vp8" [audio]]: ctor-mime: "video/webm;codecs=opus,vp8"; recording-mime: "video/webm;codecs=opus,vp8"; blob-mime: "video/webm;codecs=opus,vp8"; recorded tracks: [audio]
[MediaRecorder "video/webm;codecs=opus,vp8" [video]]: ctor-mime: "video/webm;codecs=opus,vp8"; recording-mime: "video/webm;codecs=opus,vp8"; blob-mime: "video/webm;codecs=opus,vp8"; recorded tracks: [video]
[MediaRecorder "video/webm;codecs=opus,vp8" [audio,video]]: ctor-mime: "video/webm;codecs=opus,vp8"; recording-mime: "video/webm;codecs=opus,vp8"; blob-mime: "video/webm;codecs=opus,vp8"; recorded tracks: [audio,video]
[MediaRecorder "audio/webm;codecs=opus,vp8" [audio]]: Exception; NotSupportedError: Failed to construct 'MediaRecorder': Failed to initialize native MediaRecorder the type provided (audio/webm;codecs=opus,vp8) is not supported.
[MediaRecorder "audio/webm;codecs=opus,vp8" [video]]: Exception; NotSupportedError: Failed to construct 'MediaRecorder': Failed to initialize native MediaRecorder the type provided (audio/webm;codecs=opus,vp8) is not supported.
[MediaRecorder "audio/webm;codecs=opus,vp8" [audio,video]]: Exception; NotSupportedError: Failed to construct 'MediaRecorder': Failed to initialize native MediaRecorder the type provided (audio/webm;codecs=opus,vp8) is not supported.
```

Safari: Not implemented

Given this, I think we need Part 4: Should the `MediaRecorder` `mimeType` attribute really be read-only and never change? While I do agree this makes for a simpler, less confusing, API, no implementation works this way today.


---

To recap:

1: Can the `MediaRecorder` constructor modify the `MediaRecorderOptions` `mimeType` attribute as it sets its own `mimeType` attribute?
2: Define "all the parameters" for a mime type, or rephrase. In particular, is it a MUST to include a final list of codecs?
3: What restrictions on tracks does the selected `mimeType` impose?
4: Can the `MediaRecorder` `mimeType` attribute change during the `MediaRecorder`s lifetime (to maintain behavior of current implementations)?

Please view or discuss this issue at https://github.com/w3c/mediacapture-record/issues/170 using your GitHub account

Received on Monday, 17 June 2019 16:58:42 UTC