RE: [MEDIA_PIPELINE_TF] Issues 18 and 39

From: Jan Lindquist [mailto:jan.lindquist@ericsson.com]
Sent: Monday, September 12, 2011 2:33 AM
To: Bob Lund; public-web-and-tv@w3.org
Subject: RE: [MEDIA_PIPELINE_TF] Issues 18 and 39

Hi Bob,

Below are my comments.

Best regards,
JanL

________________________________
From: Bob Lund [mailto:B.Lund@CableLabs.com]
Sent: den 8 september 2011 21:39
To: public-web-and-tv@w3.org
Cc: Jan Lindquist
Subject: RE: [MEDIA_PIPELINE_TF] Issues 18 and 39
As promised, here is my original email to Jan about Issue 18 and 39, his response and then mine.

Regards,
Bob

From: Jan Lindquist [mailto:jan.lindquist@ericsson.com]
Sent: Thursday, September 08, 2011 8:52 AM
To: Bob Lund
Cc: Clarke Stevens
Subject: RE: MPTF Issues 18 and 39

________________________________
From: Bob Lund [mailto:B.Lund@CableLabs.com]
Sent: den 7 september 2011 19:56
To: Jan Lindquist
Cc: Clarke Stevens
Subject: MPTF Issues 18 and 39
Hi Jan,

I think there is considerable overlap between your Issue 18 and my Issue 39 but there are also some differences.

Your issue 18 implementation is looking for HTML5 to expose the components of MPEG-2 TS to the application. The implementation is also looking for an API so that the application can control the components. Issue 18 is specific to MPEG-2 TS. When a new transport type comes along, e.g. MPEG-4-FF/DASH with a new organization of components then a new API may be required.

My Issue 39 identifies the set of audio, video and text track kinds that need to be exposed by the user agent for MPEG-2 TS, and other future media transports. It wants to define how the user agent does this, per track kind but it reuses the currently specified multiple audio and video track and text track APIs. Specific additions to these APIs that are required have been identified in HTML5 spec bugs http://www.w3.org/Bugs/Public/show_bug.cgi?id=13357, http://www.w3.org/Bugs/Public/show_bug.cgi?id=13358 and http://www.w3.org/Bugs/Public/show_bug.cgi?id=13359.

You have identified the need for <video> support in SVG, which I do not address.

So, we are both looking to make sure there is access to transport stream data but our approaches differ. Your's is access to the raw components while mine is through the existing APIs, and citing bugs where there is a lack in the API. It appears to me that much of what you're looking for is already supported:

Implementation: There shall be support in the video tag for an application to have the following control:
1.   An application shall retrieve a list of available components, as well as, receive events when there is a change of available components. [Exists in the multiple audio/video track and text track APIs + my bugs]
[JLI>] agree covered by bug report but not clear if requirement is covered
[BL] The bug requests an event when the set of tracks for a media resource changes. It would probably be best to list what other changes should result in notification]
[JLI2>] How do you suggest us to document this list? I am fine with listing the changes but I need to know where it will be written in order to better frame the list.
[BL2 It seems like this is a requirement so maybe we create a requirement for this in the MPTF requirements and list the specific things that need to be evented. You could also add a comment to http://www.w3.org/Bugs/Public/show_bug.cgi?id=13358. ]
2.   An application shall set and disable the active component type, as well as, receive indication of any changes on the active component type. [Exists in the multiple audio/video track and text track APIs + my bugs]
[JLI>] same as above
[BL] With regards to "set and disable", is something required beyond what the timed text track "mode" (http://dev.w3.org/html5/spec/Overview.html#text-track-model) and the audio track "enabled" and video track "selected" (http://dev.w3.org/html5/spec/Overview.html#audiotracklist-and-videotracklist-objects) provide? With regards to "receive indication ...", how is this different than the second part  of #1?]
[JLI2>] In your first reference I saw there is a "disabled" mode but the IDL indicates an "OFF" mode. They are probably the same but it should probably be clarified. I did not see an equivelent for video or audio. One could question if it is really necessary since there is mute for audio and I cannot see a use case for disabling the video, maybe there is. As to the change indication I thought the bug report covers it
[BL2 Yes, OFF is DISABLED - confusing. Equivalent for video and audio tracks are there. Video lets you choose 1 of the set - no way to select 0 or more than 1. Audio lets you select 0 or more.]
3.   An application shall set the preference order for a specific component type (ex. English, German, etc on audio languages). [Not sure what this means - display order? If so, application can do this. If UA provides controls then order up to UA]
[JLI>] There are two levels of control. A default system setting and then the default for a given instance of the video tag. The System setting agree we can leave it aside. The default during an instance needs to be looked at. We can debate is if this is really useful but if programs change overtime on the same stream don't we want to indicate the order of preference?
[BL] Text tracks have a "default" attribute. Agreed it might be useful to have similar in the case of multiple audio/video tracks]
[JLI2>]  How do you suggest how we capture this issue?
[BL2 File a bug and add to the MPTF requirements?]

4.   An application shall be able to retreive information on the currently active component per type. [Exists in the multiple audio/video track and text track APIs]
[JLI>] this is ok, no bug no issue
5.   An application shall be able to retrieve details on each component as depicted in the description section. [What details?]
[JLI>] My diagram lists a number of areas which I would like to go over.  Take AudioComponent
- PID (MPEG Program ID) : <video> label?
[BL Audio, video and text tracks have "label" attributes, although these are supposed to convey information to the user. We should discuss how these might be used]
- Language (ISO 639) : <video> lang
[BL Audio, video and text tracks have "lang" attributes]
- Component Tag (identifier of component tag) : ???
- Encoding : ???
- Encrypted : ???
In addition we have specific for video
- aspectRatio : ???
  [BL We should discuss how these would be used by Web pages]
[JLI2>] The ??? meant I did not see an equivelent. As to how they could be used is to expose information that is handled at a lower level. Understanding what aspect ratio the stream is arriving helps set the rendering aspect ratio. This will be handled natively by some default video output control of the aspect ratio. If the information is available it is possible to optimize the default video output and avoid the letter box effect.
In general I think we should allow that certain fields are exposed by the video tag.
[BL2 Agreed that the player can use them. Is there any the JavaScript could do with these values though?]
etc

6.   The video tag shall be supported for both HTML and SVG [Seems like an independent issue]
[JLI>] agree, question is how to take it forward
I think implicit in your issue is the need to do what I'm suggesting, i.e. define what components in transport streams need to be recognized by the UA. We are both addressing making this available to the application but I think the existing API suffices with some small enhancements.
Beyond my bugs, have you identified other gaps in the current API?
[JLI>] From what I could tell this is it except for possibly (5). We need to see how to handle these aspects or extensions.

Regards,
Bob Lund