Re: [media] issue-152: documents for further discussion

On Sat, May 21, 2011 at 1:30 AM, Bob Lund <B.Lund@cablelabs.com> wrote:
>
>
>> -----Original Message-----
>> From: Silvia Pfeiffer [mailto:silviapfeiffer1@gmail.com]
>> Sent: Thursday, May 19, 2011 10:12 PM
>> To: Bob Lund
>> Cc: HTML Accessibility Task Force; Mark Vickers @ Comcast; Eric
>> Winkelman; David Agranoff
>> Subject: Re: [media] issue-152: documents for further discussion
>>
>> On Thu, May 19, 2011 at 1:44 AM, Bob Lund <B.Lund@cablelabs.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Silvia Pfeiffer [mailto:silviapfeiffer1@gmail.com]
>> >> Sent: Tuesday, May 17, 2011 7:36 PM
>> >> To: Bob Lund
>> >> Cc: HTML Accessibility Task Force; Mark Vickers @ Comcast; Eric
>> >> Winkelman; David Agranoff
>> >> Subject: Re: [media] issue-152: documents for further discussion
>> >>
>> >> On Wed, May 18, 2011 at 12:44 AM, Bob Lund <B.Lund@cablelabs.com>
>> wrote:
>> >> > Hi Silvia,
>> >> >
>> >> > I had considered @data- attributes but was unsure of the
>> >> > implications
>> >> of this statement in section 3.2.3.8 of the current HTML5 spec
>> >> (http://dev.w3.org/html5/spec/Overview.html#embedding-custom-non-
>> >> visible-data-with-the-data-attributes):
>> >> >
>> >> > "User agents must not derive any implementation behavior from these
>> >> attributes or values. Specifications intended for user agents must
>> >> not define these attributes to have any meaningful values."
>> >> >
>> >> > In the case of in-band tracks, the user agent will have to create
>> >> > the
>> >> DOM equivalent of the @data- attribute for metadata tracks. This
>> >> appeared to me as being in conflict with the second sentence of the
>> >> above quote. Is this not the case?
>> >>
>> >>
>> >> Where would a UA get the information about the special track type
>> >> from from in-band metadata tracks?
>> >
>> > MPEG-2 transport streams contain program map tables that identify each
>> > program id with a type, e.g. video, audio,
>>
>> video & audio don't mean anything really.
>>
>> >EISS (http://www.cablelabs.com/specifications/OC-SP-ETV-AM1.0-I06-
>> 110128.pdf)m etc.
>>
>> I've tried to understand what EISS is. It seems to be short for
>> "Enhanced TV integrated signaling stream" and used for synchronizing an
>> application to a video program. The model behind Enhanced TV (ETV) is to
>> embed various types of data into the video stream, including programs,
>> images and triggers.
>>
>> I can tell you now that this is not how the Web works. Video is regarded
>> as a part of Web applications and not as the container for such. While I
>> can see the reasoning behind putting everything into a video container
>> and delivering it in this way to a TV, the Web works generally around
>> HTML pages and links to resources inside this HTML page that are
>> delivered in parallel and independent of the HTML page but presented
>> together with it in the Web browser. I do not see that changing any time
>> soon.
>
> Agreed. The intent is to ONLY reuse EISS (the signals to invoke the application). The delivery of the ETV application in-band is not used. URLs in the EISS messages will reference Web content that form the application, so, indeed, HTML pages and links are used just as with existing Web content.
>
> In fact, this is one of the attractions to moving to a Web model - existing Web content can be used as part of ETV applications. Whether TV or Web, there is a need to synchronize when the Web content forming the application is invoked with respect to the underlying media.
>
>>
>> In fact, my approach to a ETV signal stream would be to extract all the
>> containing information and create separate Web-conformant packages from
>> it, e.g. a common video (MPE4/WebM/Ogg format typically with one audio
>> and one video track), separate image resources, separate Web pages, and
>> separate caption, advertising etc tracks. Then we can go back to the
>> known way of delivering content on the Web.
>
> That is the model, see above. But there is still the need for a signal so the Web content can be synchronized with the media stream.


You can use MutableTextTracks for that.


>> >MPEG-2 TS may be used directly over HTTP or as the fragment format in
>> HTTP live streaming and DASH.
>> >
>> >> Do you know fields in MP4, Ogg and WebM that provide such
>> >> information?
>> >
>> > Fragmented MP4 will be carried in some adaptive bit-rate containers,
>> e.g. certain DASH profiles. In this case, the metadata tracks will be
>> identified in the manifest file. However, with respect to the HTML5
>> "timed text track" API these are still in-band, i.e. not sourced from an
>> external file. In this case, there is still the need to identify the
>> type of metadata. Discussions are taking place now in MPEG and other
>> places regarding requirements for identifying metadata tracks in DASH.
>>
>>
>> I agree the HTTP adaptive streaming may create a use case where we have
>> to deal with a complex resource of potentially many tracks.
>> However, we haven't even decided on how to solve HTTP adaptive streaming
>> in the browser yet.
>
> While there may be aspects of HTTP adaptive streaming that make sense to expose in HTML, this does not need to be done to solve HTTP adaptive streaming in a browser. We've added browser support for HTTP Live Streaming by adding a module to the media pipeline that understands the HTLS manifest file. The <video> src points to the manifest file URL and there is no impact to HTML.


The only browser that will do something with this manifest is Safari
on all devices (including iOS). That's not "added browser support" -
that's just a special implementation. The other way of doing it is
through a browser plugin, in which case you can do whatever you want
with flags etc. We have no standard for HTTP adaptive streaming that
is implemented across multiple browser, which is what HTML and the API
is about.


>> Many discussions are going on about this right now
>> in several forums. Right now, I cannot see which of the existing
>> solutions would become adopted by the browsers or whether it may even be
>> a new one. My gut feeling is that the functionality may be a subpart of
>> DASH, even though DASH itself may not be able to be picked for IP
>> reasons. So, until we know what will happen there, let's not create a
>> solution for something that hasn't been decided on yet.
>
> No matter how adaptive streaming is ultimately exposed in HTML, the issues I'm raising with how metadata gets exposed will need to solved. The solution I'm proposing will work with MPEG-2 TS, DASH and Microsoft Smooth Streaming. It would also work with HLS if text tracks were supported in the manifest file.


So, my key issue with this is that you are assuming that it makes
sense to put application-specific text tracks into the manifest. I
don't subscribe to this. If we don't put application-specific tracks
inside MP4, Ogg and WebM, then they also don't make sense in a
manifest and should be exposed through <track> directly to the video
element.


>> As for the inclusion of metadata tracks in HTTP adaptive streaming
>> manifests: right now my thinking is that it makes no sense to include
>> text tracks into a manifest, because the manifest has the sole purpose
>> to switch between tracks of differing bitrate for video (and maybe for
>> audio). Since text tracks are typically of a much smaller bandwidth that
>> audio or video and contain very concise information that cannot be "bit
>> peeled", they should not take part in the adaptive manifest.
>> Instead, they should be delivered through the <track> element.
>
> Support for text tracks are specified in the DASH spec. See http://lists.w3.org/Archives/Public/public-html-a11y/2011Feb/0151.html example 1 in appendix G.

Sure. But DASH is not supported in HTML. I am saying we need to wait
until that time. The a11y TF is not the right forum to discuss
addition of HTTP adaptive streaming to HTML.


>> But let's wait for a specification and for some trial implementations on
>> this. I think we are trying to solve a problem that doesn't even exist
>> yet.
>>
>>
>> >> If there is such a field that you need exposed on top of what is
>> >> already there, then it would indeed make sense to include that.
>> >
>> > As described above, there is or will be such a field.
>> >
>> >> But I honestly doubt that you will find in-band tracks that will tell
>> >> you that they contain adinsertion information or syncwebcontent data.
>> >
>> > See above.
>>
>>
>> I'm still skeptical about application-specific metadata tracks inside a
>> MP4/WebM/Ogg file. At this stage I'd say: show me an example file that
>> has this and some software that exists now and that extracts it and a
>> use case where these need to be on the Web.
>
> With regards to examples:
>
> - The link above shows text tracks in DASH
> - CableLabs, on behalf of its cable company members, has submitted comments to the DASH DIS to add metadata support.
>
> - To quote David Singer in his previous email on this thread:
>
> " I may be getting into the middle of something I don't understand, but...
>
> * we can easily add a metadata tag to MP4 tracks to declare their kind/role
> * MP4 has metadata tracks, that could be used to carry any format (it just needs defining)


It doesn't just need defining in MP4 and DASH - it also needs defining
in the HTTP adaptive streaming approach to be adopted for HTML and in
Ogg and in WebM to get the broadest possible cover. Just saying that
the problem may be larger and depend on more steps that you might
think.


> * we can use caption track events with a caption track that has no visible text (we may need a tag for the 'cookie')"
>
> - My previous email has identified 3 types of application metadata that is carried in MPEG-2 TS today.


You identified "audio" and "video" and "EISS". Only one of them is
application-specific metadata and EISS is not just one track, but a
whole application suite that you agreed makes more sense if unpacked
server-side. EISS would only need a signal track, which can be done
with MutableTextTrack for live and with <track> for non-live. I don't
see a need that is not met.


> With regards to use cases, North American cable operators have identified the requirement that existing application metadata applications continue to work with browser based clients.


Good, so let's try and find a way to make them work. Let's try to do
that in JavaScript for now. Once we hit a barrier with that, we have a
case to make.


> Perhaps this should be a topic for one of the weekly media group calls.

Is this about accessibility? From what I've seen this far, I don't
think this is the case and as such this isn't the right forum. In
fact, I think the Web and TV forum is the much more appropriate group
for this discussion.

Cheers,
Silvia.

Received on Saturday, 21 May 2011 09:07:52 UTC