W3C home > Mailing lists > Public > public-tt@w3.org > July 2014

Re: ISSUE-317 (IMSC should not require frame alignment): IMSC should not require frame alignment [TTML IMSC 1.0]

From: John Birch <John.Birch@screensystems.tv>
Date: Thu, 10 Jul 2014 08:09:47 +0000
To: "'pal@sandflow.com'" <pal@sandflow.com>, "'nigel.megitt@bbc.co.uk'" <nigel.megitt@bbc.co.uk>
CC: "'public-tt@w3.org'" <public-tt@w3.org>
Message-ID: <0981DC6F684DE44FBE8E2602C456E8AB016C0B32D5@SS-IP-EXMB-01.screensystems.tv>
I've tackled this misconception before. Using frame identifiers is not the only mechanism. The offset from the start of media (expressed in any timebase - including SMPTE formats) also works - this is media timebase.

Other synchronisation systems are also possible and in use - including wall clock (yeuck), synchronisation using audio fingerprints, synchronisation using video watermarks and synch using other stream data / markers.

These might be out of scope for TTML / IMSC, but Nigel has a compelling use case... There is no guarantee that there will be anything like a SMPTE timecode in the distributed media. In most cases the SMPTE timing reference will be 'synthesised' in the player from other mechanisms.

Further there is no requirement that the time expressions in the subtitle file have a timebase that matches that of the media... A timecode represents both a frame AND a point along a timeline (that at distribution can be assumed as continuous). One second in one timebase is one second in any other. Only the reolution of timing is affected by timebase.

The above of course excludes scenarios wjere video is played deliberately fast (e.g. 24 at 25) or slow.

In fact it is arguably better to use media timing in distribution and 'preload' all these timebase issues at the server.

Best regards,
John

John Birch | Strategic Partnerships Manager | Screen
Main Line : +44 1473 831700 | Ext : 2208 | Direct Dial : +44 1473 834532
Mobile : +44 7919 558380 | Fax : +44 1473 830078
John.Birch@screensystems.tv | www.screensystems.tv | https://twitter.com/screensystems

Visit us at
IBC, RAI Amsterdam, 12-16 Sept, stand 1.C49

P Before printing, think about the environment----- Original Message -----
From: Pierre-Anthony Lemieux [mailto:pal@sandflow.com]
Sent: Wednesday, July 09, 2014 11:51 PM
To: Nigel Megitt <nigel.megitt@bbc.co.uk>
Cc: Timed Text Working Group <public-tt@w3.org>
Subject: Re: ISSUE-317 (IMSC should not require frame alignment): IMSC should  not require frame alignment [TTML IMSC 1.0]

Hi Nigel,

Mapping media time expression to video frames is to only means of
allowing authors to unambiguously specify on which frame of the
related video object a subtitle will appear.

If a system chooses to decimate the video for distribution, i.e. after
the content has been authored and QCed, the system can simply specify
that the video be played back at the original frame rate, at least for
the purposes of subtitle/caption rendering. In other words, the
related video object at playback will have the same frame rate as
during authoring.

Best,

-- Pierre

On Fri, Jun 20, 2014 at 11:34 AM, Nigel Megitt <nigel.megitt@bbc.co.uk> wrote:
> Hi Pierre,
>
> Comments in line:
>
> On 29/05/2014 07:20, "Pierre-Anthony Lemieux" <pal@sandflow.com> wrote:
>
>>Hi Nigel,
>>
>>As I understand it, the motivation to map media time expression to
>>video frames is to allow authors to unambiguously determine on which
>>frame of the related video object a subtitle will appear. I would
>>think this is particularly important on scene boundaries.
>
> I formerly was of that opinion too; however I now believe, from user
> feedback, that this is not a strong enough concern to create a design
> constraint for. Certainly it is less important than maintaining a maximum
> reading speed and equivalent flow of content to the audio to which
> subtitles and captions. Maximum reading speed is a matter of regulation in
> the UK.
>
>>This unambiguous determination is not possible if the time expression
>>mapping depends on the display frame rate, which is not known at time
>>of authoring and largely arbitrary.
>>
>>I would expect the timed text content to remain usable in the current
>>approach as long as the duration of intermediate synchronic documents
>>duration remains larger than the frame duration. Is there a use case
>>where this would systematically not be the case?
>
> I do not believe that such a rule would achieve the requirements. Even
> with ISD durations that are longer than frame duration quantisation
> effects could in principle reduce the time available for reading each
> word, especially for low frame rates. I would consider 6 frames per second
> to be potential and realistic targets for distribution of video that goes
> alongside captions and subtitles.
>
>>In general, without more information on the use case below (e.g. what
>>is the minimum frame rates/why not use reduced resolution instead, how
>>frequently will this frame rate reduction happen, is IMSC1 even
>>applicable/desirable...), I am reluctant to suggest a change to an
>>approach that appears to be working for users.
>
> Reduced resolution is a comparable but separate topic.
>
> Kind regards,
>
> Nigel
>
>
>>
>>Best,
>>
>>-- Pierre
>>
>>On Tue, May 27, 2014 at 4:07 AM, Nigel Megitt <nigel.megitt@bbc.co.uk>
>>wrote:
>>> Hi Pierre,
>>>
>>>>>Yes, they are able to reduce frame rate as an available
>>>>>option for adapting to network conditions.
>>>>
>>>>Is the BBC doing this? If not, who? Just trying to fully understand
>>>>the use case.
>>>
>>> You're hitting my limits on how much I can freely discuss on this forum.
>>> Reduced frame rate distribution options are likely to be used in the way
>>> I've described by organisations you've heard of within the lifetime of
>>> IMSC1. Knowing who won't enhance the use case further.
>>>
>>> Kind regards,
>>>
>>> Nigel
>>>
>>>
>>>
>>> On 23/05/2014 17:46, "Pierre-Anthony Lemieux" <pal@sandflow.com> wrote:
>>>
>>>>Hi Nigel,
>>>>
>>>>> Yes, they are able to reduce frame rate as an available
>>>>> option for adapting to network conditions.
>>>>
>>>>Is the BBC doing this? If not, who? Just trying to fully understand
>>>>the use case.
>>>>
>>>>Thanks,
>>>>
>>>>-- Pierre
>>>>
>>>>On Fri, May 23, 2014 at 9:24 AM, Nigel Megitt <nigel.megitt@bbc.co.uk>
>>>>wrote:
>>>>> On 23/05/2014 17:20, "Pierre-Anthony Lemieux" <pal@sandflow.com>
>>>>>wrote:
>>>>>
>>>>>>> Yes - it's likely that adaptive streaming distribution mechanisms
>>>>>>> will do this.
>>>>>>
>>>>>>Do they do this today?
>>>>>
>>>>> Yes, they are able to reduce frame rate as an available option for
>>>>> adapting to network conditions.
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Nigel
>>>>>
>>>>>>
>>>>>>Best,
>>>>>>
>>>>>>-- Pierre
>>>>>>
>>>>>>On Fri, May 23, 2014 at 9:19 AM, Nigel Megitt <nigel.megitt@bbc.co.uk>
>>>>>>wrote:
>>>>>>> Hi Pierre,
>>>>>>>
>>>>>>>>Hi Nigel,
>>>>>>>>
>>>>>>>>> 2. Video encoded for distribution at 7.5fps.
>>>>>>>>
>>>>>>>>Can you point to actual use case/deployment for this?
>>>>>>>
>>>>>>> Yes - it's likely that adaptive streaming distribution mechanisms
>>>>>>>will
>>>>>>>do
>>>>>>> this. Profiles for this purpose that go down to e.g. 25/4=6.25 fps.
>>>>>>>are
>>>>>>> likely to be used for mobile devices for example. I assume that
>>>>>>>something
>>>>>>> similar will result from a starting point of 30fps.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> Nigel
>>>>>>>
>>>>>>>>On Fri, May 23, 2014 at 9:11 AM, Nigel Megitt
>>>>>>>><nigel.megitt@bbc.co.uk>
>>>>>>>>wrote:
>>>>>>>>> Hi Pierre,
>>>>>>>>>
>>>>>>>>> It would certainly help to explain the model more clearly, along
>>>>>>>>>the
>>>>>>>>>lines
>>>>>>>>> that you've outlined. The specific proposal wouldn't help though,
>>>>>>>>>since
>>>>>>>>> the precise timing information would have been lost at the point
>>>>>>>>>of
>>>>>>>>> temporal quantisation and could not be regenerated later.
>>>>>>>>>
>>>>>>>>> For example this suggests a chain such as:
>>>>>>>>>
>>>>>>>>> 1. IMSC Document authored against video at 30fps.
>>>>>>>>> 2. Video encoded for distribution at 7.5fps.
>>>>>>>>> 3. Receiving system must align the resolved TTML time expressions
>>>>>>>>>with
>>>>>>>>>the
>>>>>>>>> 7.5fps 'quanta' prior to display, as per the rule in the current
>>>>>>>>>spec.
>>>>>>>>>
>>>>>>>>> Even if the display device refreshes at 60fps it would be
>>>>>>>>>forbidden
>>>>>>>>>from
>>>>>>>>> using the original timings because the spec references the encoded
>>>>>>>>>video.
>>>>>>>>>
>>>>>>>>> What I'm trying to get to is a solution that is permitted
>>>>>>>>>(actually
>>>>>>>>> encouraged) to align with display frames as late as possible while
>>>>>>>>>losing
>>>>>>>>> minimal information. In some real world systems that's unavoidably
>>>>>>>>>earlier
>>>>>>>>> than the display, but we shouldn't use the lowest common
>>>>>>>>>denominator
>>>>>>>>>to
>>>>>>>>> set the rule for all implementations.
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>>
>>>>>>>>> Nigel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 23/05/2014 16:33, "Pierre-Anthony Lemieux" <pal@sandflow.com>
>>>>>>>>>wrote:
>>>>>>>>>
>>>>>>>>>>Hi Nigel,
>>>>>>>>>>
>>>>>>>>>>IMSC 1.0 4.4 [1] refers to synchronization with the related video
>>>>>>>>>>object against which the timed text content is delivered, not
>>>>>>>>>>synchronization to the displayed frame rate by the
>>>>>>>>>>terminal/UA/device/display/TV. In other words, if a
>>>>>>>>>>terminal/UA/device/display/TV chooses to alter the video frame
>>>>>>>>>>rate
>>>>>>>>>>of
>>>>>>>>>>the related video object it receives (for whatever reason), then I
>>>>>>>>>>expect it will accordingly alter the timed text display (perhaps
>>>>>>>>>>along
>>>>>>>>>>the lines of what is suggested below), with the knowledge that the
>>>>>>>>>>timed text was authored according to the constraints of Section
>>>>>>>>>>4.4.
>>>>>>>>>>
>>>>>>>>>>Would a note to that effect help?
>>>>>>>>>>
>>>>>>>>>>Thanks,
>>>>>>>>>>
>>>>>>>>>>-- Pierre
>>>>>>>>>>
>>>>>>>>>>On Thu, May 22, 2014 at 3:19 AM, Timed Text Working Group Issue
>>>>>>>>>>Tracker <sysbot+tracker@w3.org> wrote:
>>>>>>>>>>> ISSUE-317 (IMSC should not require frame alignment): IMSC should
>>>>>>>>>>>not
>>>>>>>>>>>require frame alignment [TTML IMSC 1.0]
>>>>>>>>>>>
>>>>>>>>>>> http://www.w3.org/AudioVideo/TT/tracker/issues/317
>>>>>>>>>>>
>>>>>>>>>>> Raised by: Nigel Megitt
>>>>>>>>>>> On product: TTML IMSC 1.0
>>>>>>>>>>>
>>>>>>>>>>> IMSC 1.0 4.4 [1] currently requires temporal quantisation of
>>>>>>>>>>>media
>>>>>>>>>>>times to frame display times. This rule comes into play when
>>>>>>>>>>>times
>>>>>>>>>>>are
>>>>>>>>>>>not expressed in frames, and therefore the same document may
>>>>>>>>>>>apply
>>>>>>>>>>>to a
>>>>>>>>>>>range of related media objects covering different frame rates. In
>>>>>>>>>>>the
>>>>>>>>>>>case when frames are used the document can only be displayed
>>>>>>>>>>>alongside
>>>>>>>>>>>media of the same frame rate so there's no need for the frame
>>>>>>>>>>>alignment
>>>>>>>>>>>expression.
>>>>>>>>>>>
>>>>>>>>>>> This approach prevents implementations from changing caption
>>>>>>>>>>>display
>>>>>>>>>>>at
>>>>>>>>>>>screen refresh rate quantisation and enforces quantisation based
>>>>>>>>>>>on
>>>>>>>>>>>the
>>>>>>>>>>>encoded video frame rate. This means that if a low frame rate
>>>>>>>>>>>video
>>>>>>>>>>>is
>>>>>>>>>>>provided, e.g. quarter rate which could be around 6 frames per
>>>>>>>>>>>second,
>>>>>>>>>>>the effective word reading rate may be increased to the point
>>>>>>>>>>>where
>>>>>>>>>>>text
>>>>>>>>>>>becomes hard to read.
>>>>>>>>>>>
>>>>>>>>>>> Consider a streaming environment in which there is enough
>>>>>>>>>>>network
>>>>>>>>>>>capacity to provide audio and captions but the video experience
>>>>>>>>>>>is
>>>>>>>>>>>badly
>>>>>>>>>>>impacted: in this case it must be permitted that the
>>>>>>>>>>>implementation
>>>>>>>>>>>continue to present captions alongside the audio regardless of
>>>>>>>>>>>the
>>>>>>>>>>>frames of video that are displayed.
>>>>>>>>>>>
>>>>>>>>>>> I propose a solution to this problem that implementations SHALL
>>>>>>>>>>>display
>>>>>>>>>>>captions as temporally close to the media time specified as the
>>>>>>>>>>>display
>>>>>>>>>>>device permits, independent of video frame rate.
>>>>>>>>>>>
>>>>>>>>>>> Note that where frames are used in media time expressions this
>>>>>>>>>>>reduces
>>>>>>>>>>>to exactly the current behaviour.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>>https://dvcs.w3.org/hg/ttml/raw-file/ea1a92310a27/ttml-ww-profile
>>>>>>>>>>>s/
>>>>>>>>>>>tt
>>>>>>>>>>>ml
>>>>>>>>>>>-w
>>>>>>>>>>>w-profiles.html#synchronization
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----------------------------
>>>>> http://www.bbc.co.uk
>>>>> This e-mail (and any attachments) is confidential and
>>>>> may contain personal views which are not the views of the BBC unless
>>>>>specifically stated.
>>>>> If you have received it in
>>>>> error, please delete it from your system.
>>>>> Do not use, copy or disclose the
>>>>> information in any way nor act in reliance on it and notify the sender
>>>>> immediately.
>>>>> Please note that the BBC monitors e-mails
>>>>> sent or received.
>>>>> Further communication will signify your consent to
>>>>> this.
>>>>> -----------------------------
>>>
>>>
>>>
>>> -----------------------------
>>> http://www.bbc.co.uk
>>> This e-mail (and any attachments) is confidential and
>>> may contain personal views which are not the views of the BBC unless
>>>specifically stated.
>>> If you have received it in
>>> error, please delete it from your system.
>>> Do not use, copy or disclose the
>>> information in any way nor act in reliance on it and notify the sender
>>> immediately.
>>> Please note that the BBC monitors e-mails
>>> sent or received.
>>> Further communication will signify your consent to
>>> this.
>>> -----------------------------
>


This message may contain confidential and/or privileged information. If you are not the intended recipient you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. Screen Subtitling Systems Ltd. Registered in England No. 2596832. Registered Office: The Old Rectory, Claydon Church Lane, Claydon, Ipswich, Suffolk, IP6 0EQ
Received on Thursday, 10 July 2014 08:10:20 UTC

This archive was generated by hypermail 2.3.1 : Thursday, 5 October 2017 18:24:16 UTC