Re: HTML5 support for track metadata

On 11 Mar 2014 08:21, "Bob Lund" <B.Lund@cablelabs.com> wrote:
>
>
>
> On 3/9/14, 11:50 PM, "Silvia Pfeiffer" <silviapfeiffer1@gmail.com> wrote:
>
> >Hi Bob & Clift,
> >
> >I have actually wondered about the need for exposing in-band track
> >metadata here myself
> >
https://www.w3.org/community/inbandtracks/wiki/Main_Page#Exposing_In-band_
> >Track_Metadata
> >.
> >
> >I believe the biggest discussion that was had here wrt this topic is
> >to expose the PMT. I agree with Clift and Bob that exposing the PMT in
> >a separate track is not necessary, since the list of tracks in the
> >media element (.audioTracks, .videoTracks and .textTracks) as well as
> >the attributes on the track objects provide sufficient information
> >about the PMT.
> >
> >Thus, we have to discuss whether there is a use case for providing a
> >generic means for exposing other in-band metadata text tracks as
> >specified here:
> >
https://www.w3.org/community/inbandtracks/wiki/Main_Page#Guidelines_for_cr
> >eating_metadata_text_track_cues
> >
> >What other tracks does a MPEG-2 TS have that are not audio, video,
> >captions/subtitles, or the PMT?
>
> The other track types are: region_rating_table used in parental controls,
> SCTE 35 used for ad insertion, EISS as in enhanced TV. I have been told by
> Nielsen rating that they want to embed proprietary data in a private user
> track to support collecting information on Web TV platforms.

Sounds reasonable. I think that's what metadata tracks are made for, since
this information seems to be timed metadata. I'd just map this to DataCue.

HTH.

Cheers,
Silvia.

>
> >
> >Cheers,
> >Silvia.
> >
> >
> >
> >
> >On Wed, Mar 5, 2014 at 9:11 AM, Bob Lund <B.Lund@cablelabs.com> wrote:
> >>
> >>
> >> From: <Clift>, Graham <Graham.Clift@am.sony.com>
> >> Date: Tuesday, March 4, 2014 at 2:17 PM
> >> To: Bob Lund <b.lund@cablelabs.com>, "public-inbandtracks@w3.org"
> >> <public-inbandtracks@w3.org>
> >> Cc: "Ota, Takaaki" <Takaaki.Ota@am.sony.com>, "Wu, Max"
> >> <Max.Wu@am.sony.com>, "Nejat, Mike" <Mahyar.Nejat@am.sony.com>, Brant
> >> Candelore <brant.candelore@am.sony.com>
> >> Subject: RE: HTML5 support for track metadata
> >>
> >> Hi CG,
> >>
> >>
> >>
> >> I believe that option 3 is the best approach because it requires least
> >> change to HTML5 specification, covers the use cases and will be the
most
> >> likely to pass the HTML WG.
> >>
> >>
> >>
> >> As to how to handle events (with minimal change to HTML spec), I was
> >> thinking the following:
> >>
> >>
> >>
> >> When an application sees a change in the PMT there should also be a
> >> corresponding change to one or more of the
> >> TextTrackList/VideoTrackList/AudioTrackList 's.
> >>
> >> Seems then it should be sensible to tie the PMT change in
> >> inBandMetaDataDispatchType to an onchange event in the TextTrackList
> >>object.
> >> This would be sufficient for the purpose tracking PMT data and if cues
> >>are
> >> needed then they could be generated easily by the application thus
> >> eliminating the only advantage that option 1 has IMHO.
> >>
> >>
> >> Changes in the PMT caused by adding/removing elementary streams should
> >> result in one of onhange, onddrack, onemovetrack events. Are there any
> >>use
> >> cases where the PMT for an existing track changes?
> >>
> >>
> >>
> >> BTW,
> >>
> >>
> >>
> >> As well as deciding how to handle the PMT data via the three
> >>alternatives I
> >> believe more work should be done on Item 3) (creating in-band metadata
> >>text
> >> tracks  cues). In particular there needs to be more clarification to
> >>ensure
> >> consistency across implementations. The two areas below is where I see
> >>some
> >> problems:
> >>
> >>
> >>
> >> a)       Handling private section payloads that are split across many
TS
> >> packets is not well defined.
> >>
> >>
> >> My proposal defines this - a complete private section is returned in
the
> >> cue.
> >>
> >> Seems there are three possible approaches.
> >>
> >>                      i.                              Since the
detection
> >> method proposed is the 'payload_unit_start_indicator', this would
> >>suggest
> >> that the transport demux is collecting the individual payloads before
> >> presenting them as a cue to the web application. If this is the case
> >>then
> >> how does the demux decide that the payload is complete? If it waits to
> >>see
> >> the next 'payload_unit_start_indicator' then the timing may be too late
> >>to
> >> be relevant.
> >>
> >>                    ii.                              If, on the other
> >>hand,
> >> the demux creates a cue for each payload entity then that could impact
> >> performance.
> >>
> >>                   iii.                              Maybe it just waits
> >>for
> >> a period of time and if no more payloads packets are received then cue
> >>what
> >> we have. This approach would mean there is an expectation for the
> >> application to handle variably fragmented cues.
> >>
> >>
> >>
> >> Leaving it up the UA to decide how to implement could be challenging
> >>for web
> >> application designers to allow for all possiblities.
> >>
> >> b)      There is no clarity of what the startTime means to the
> >>application.
> >> The spec says this is with respect to the media resource time which
> >> presumably means the PTS from some audio/video PES. But which timing?
> >>After
> >> the private payload is sent or before private payload is sent? And what
> >>is
> >> the startTime for payloads split across TS packets with Video PES
> >>packets in
> >> between, especially when partial payload DataCues are supported as  (as
> >>in
> >> case iii above.) The option of leaving this up to the implementation is
> >> unsatisfactory because of the potential for variations in the result.
> >>
> >>
> >> I agree more guidelines in cue generation would remove ambiguity. I
> >>think
> >> startTime in the case of MPEG-2 TS metadata should be the current time
> >>in
> >> the media resource when the private section is received.
> >>
> >>
> >>
> >> Regards
> >>
> >>
> >>
> >> Graham Clift
> >>
> >>
> >>
> >>
> >>
> >> From: Bob Lund [mailto:B.Lund@CableLabs.com]
> >> Sent: Tuesday, March 04, 2014 11:50 AM
> >> To: public-inbandtracks@w3.org
> >> Subject: HTML5 support for track metadata
> >>
> >>
> >>
> >> Hi CG,
> >>
> >>
> >>
> >> I think that the existing HTML5 CR spec [1] is very close to supporting
> >>the
> >> use cases that are being discussed in the CG. I propose submitting
> >>several
> >> HTML5 bugs to close the gap and I'm interested in your thoughts.
> >>
> >>
> >>
> >> An alternative described in the CG wiki is to use kind and
> >> inBandMetadataTrackDispatchType attributes to expose track metadata:
> >>kind is
> >> used for all tracks except metadata text tracks and
> >> inBandMetadataTrackDispatchType is used for metadata text tracks. This
> >> covers the use cases that have been described in the CG but requires
> >>some
> >> additions to existing HTML5 CR sections:
> >>
> >>
> >>
> >> 1) Additions to the table "Return values for AudioTrack.kind() and
> >> VideoTrack.kind()" in [2] describing how @kind should be set for
various
> >> track types. [3] shows the new additions for MPEG-2 TS and DASH media
> >> resources.
> >>
> >>
> >>
> >> 2) Text track equivalent of the table "Return values for
> >>AudioTrack.kind()
> >> and VideoTrack.kind()" in [2]. [4] shows such an equivalent table for
> >> setting @kind for text tracks in MPEG-2 TS and DASH. This could go in
> >>the
> >> HTML5 CR spec here [5].
> >>
> >>
> >>
> >> 3) Guidelines for creating in-band metadata text track cues. Here is
the
> >> start for MPEG-2 TS [5]. This table could go here [6].
> >>
> >>
> >>
> >> 4) Additional definition for DASH describing how to set
> >> inBandMetadataTrackDispatchType[7].
> >>
> >>
> >>
> >> Does anyone see a reason not to file bugs to add 1-4 above? These
> >>changes
> >> are consistent with the direction already taken in [1]. Making these
> >>changes
> >> wouldn't preclude further work in the CG and would address use cases
> >>that
> >> have been identified so far.
> >>
> >>
> >>
> >> Bob
> >>
> >>
> >>
> >> [1] http://www.w3.org/TR/html5/
> >>
> >> [2]
> >>
> >>
http://www.w3.org/TR/html5/embedded-content-0.html#audiotracklist-and-vid
> >>eotracklist-objects
> >>
> >> [3]
> >>
> >>
https://www.w3.org/community/inbandtracks/wiki/Main_Page#Audio_and_video_
> >>kind_table
> >>
> >> [4]
> >>https://www.w3.org/community/inbandtracks/wiki/Main_Page#Text_kind_table
> >>
> >> [5]
> >>
> >>
https://www.w3.org/community/inbandtracks/wiki/Main_Page#Guidelines_for_c
> >>reating_metadata_text_track_cues
> >>
> >> [6]
> >>
> >>
http://www.w3.org/TR/html5/embedded-content-0.html#sourcing-in-band-text-
> >>tracks
> >>
> >> [7]
> >>
> >>
https://www.w3.org/community/inbandtracks/wiki/Main_Page#Exposing_a_Media
> >>_Resource_Specific_TextTrack
> >>
> >>
>