SMIL 2.0 Extension for Professional Multimedia Authoring - Preliminary Investigation

Yoshihisa Gonno

ygonno@sm.sony.co.jp

Sony Corporation

2003.2.14

Outline

SMIL: Current Status

SMIL: Key to Success in Business

Innovation: Media Authoring Process

Requirements for Professional Media Authoring Description Language

SMIL 2.0

Professional Media Clipping

Requirement:
Temporal Duration Media Clipping
  • media clipping capability by referring embedded SMPTE Time Codes (TCs)
Limitation:
  • MediaClipping Module says:

    The clipBegin attribute specifies the beginning of a sub-clip of a continuous media object as offset from the start of the media object. This offset is measured in normal media playback time from the beginning of the media.

Proposal:
  • If the media have embedded time codes, the offset should be measured with the embedded time codes.
See next slide
Affect:
Media Objects Modules (Re-definition/Clarification)

Professional Media Clipping

				CASE1:
<ref src="media1" clipBegin="smpte=01:02:00:00" clipEnd="smpte=01:06:00:00"/>


media1 |---------+++++++++++++++++++++---------|           8-minute-long
      01:00:..  01:02:00:00         01:06:..  01:08:00:00  embedded TCs in media1
ref              |+++++++++++++++++++|                     4-minute-long
                00:00:00:00         00:04:00:00


				CASE2:
<ref src="media2" clipBegin="smpte=00:02:00:00" clipEnd="smpte=00:06:00:00"/>


media2     empty          //   |-------------------|           8-minute-long
       00:02:..  00:06:.. //  01:00:00:00         01:08:00:00  embedded TCs in media2
ref     |---------|       //                                   4-minute-long
       00:00:..  00:04:.. //

Media Type Casting

Requirement:
Media Track Identification
  • extracting and handling individual types of media, e.g. video, audio, out of complex media
Limitation:
  • BasicMedia Module says:

    All of these media elements are semantically identical. When playing back a media object, the player must not derive the exact type of the media object from the name of the media object element.

  • How can we get media1 video with media2 audio ?
Media Type Casting
Proposal:
  • video element should represent video, audio element should represent audio, exclusively.
<par>
  <video src="media1"/>
  <audio src="media2"/>
</par>
Affect:
Media Objects Modules (Re-definition/Clarification)

Media Track Identification and Destination

Requirement:
Media Track Identification
  • manipulating individual tracks within a media material
    • a media material may consist of multiple tracks
Proposal:
						by XPointer-like URI fragment:
<audio src="media1#track('A1')" cast="left"/>

by attribute:
<audio src="media1" track="A1" cast="left"/>
Media Track Identification
Affect:
Media Objects Modules (Extension)

Visual Area Clipping

Requirement:
Spatial Region Media Clipping
  • media clipping capability by specifying coodinates
Proposal:
  • apply coords attribute to media object elements and time container elements
<video src="media2" coords="0%,0%,50%,100%"/>

<seq coords="50%,0%,100%,100%">
  <video src="media1"/>
  <video src="media2"/>
</seq>
Affect:
Media Objects Modules, Timing and Synchronization Modules (Extension)

Media Transparency Modification

Requirement:
Visual Media Transparency (Alpha-Blending)
Proposal:
  • apply alpha attribute to media object elements and time container elements
<par alpha="0.5">
  <video src="media1" alpha="0.8"/>
  <video src="media2" alpha="0.2"/>
</par>
Affect:
Media Objects Modules, Timing and Synchronization Modules (Extension)

Media Time Addressing

Requirement:
Media Time Identification
  • embedding Time Codes (TCs) into resulting media
Proposal:
  • apply SMPTE TCs to begin|end attributes
See next slide
Affect:
Timing and Synchronization Modules (Extension)

Media Time Addressing

<ref src="media1" begin="smpte=01:00:00:00" end="smpte=01:05:00:00"/>

  media1 |+++++++++++++++++++|
        03:00:00:00         03:04:00:00  embedded TCs in media1
        01:00:00:00         01:04:00:00  applied TCs to ref element


<seq begin="smpte=01:00:00:00">
  <ref src="media1"/>
  <ref src="media2"/>
</seq>

  media1 |+++++++++++++++++++|
        03:00:00:00         03:04:00:00 embedded TCs in media1
  media2                     |+++++++++++++++++++|
                            02:00:00:00         02:04:00:00 embedded TCs in media2
  seq    |+++++++++++++++++++++++++++++++++++++++|
        01:00:00:00         01:04:00:00         01:08:00:00 applied TCs to seq element

Time Container Clipping

Requirement:
Intermediate Media Identification
  • improvement of Time Container (par|seq) usability as Media Object
Proposal:
  • apply clipBegin|clipEnd attributes to time container elements
See next slide
Affect:
Timing and Synchronization Modules (Extension)

Time Container Clipping

<ref src="media0" clipBegin="smpte=01:02:00:00" clipEnd="smpte=01:06:00:00"/>

  Given media0 as a par element below:

    <par>
      <ref src="media1" begin="smpte=01:00:00:00"/>
      <ref src="media2" begin="smpte=01:03:00:00"/>
    </par>


to be equivalent to a par element below:

<par clipBegin="smpte=01:02:00:00" clipEnd="smpte=01:06:00:00">
  <ref src="media1" begin="smpte=01:00:00:00"/>
  <ref src="media2" begin="smpte=01:03:00:00"/>
</par>

Metadata Description

Requirement:
Metadata Description
  • inclusion of non-RDF XML descriptions
    • e.g. MPEG-7 Descriptions
    • may be introduced with rdf:parseType="Literal", but this makes descriptions unnecessarily redundant
Proposal:
  <head>
    <metadata type="text/xml">
      <mp7:mpeg7 xmlns:mp7="urn:mpeg:mpeg7:...">
        ....
      </mp7:mpeg7>
    </metadata>
    <metadata type="application/rdf+xml">
      <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
        ....
      </rdf:RDF>
    </metadata>
  </head>
Affect:
Metainformation Module (Extension)

Metadata Synchronization

Requirement:
Metadata Synchronization
  • metadata synchronization with audio/video streams
    • e.g. MPEG-7 Binary Stream
Proposal:
  • using ref element is possible but ambiguous
  • should define another Media Object element ?
    • audio|video|img|animation|text|textstream
  • why not for metadata
    • meta|metadata elements are already in use for other purpose
    • metastream ?
Affect:
Media Objects Modules (Extension)

Media Transition

Requirement:
Transition Effects on Time Container Elements
  • improvement of Time Container (par|seq) usability as Media Object
Proposal:
  • allow transIn|transOut attributes to be applied to Time Container elements
Why not ?
<par transOut="crossfade">
  <ref src="media1"/>
  <ref src="media2"/>
</par>
rather than
<par>
  <ref src="media1" transOut="crossfade"/>
  <ref src="media2" transOut="crossfade"/>
</par>
Affect:
Transition Effects Modules (Extension)

Audio Transition

Requirement:
Audio Transition
  • only visual behavior is defined, what about audio and other data ?
  • should be clearly defined
Proposal:
  • increase/decrease soundLevel along with transition progress
  • needs more consideration
Affect:
Transition Effects Modules (Re-definition/Clarification)

Transformation Effects Modules

Requirement:
Media Composition and Transformation
Proposal:
  • follow the syntax of Transition Effects Modules
    BasicTransformations Module
    declare transformation element in head element,
    refer with transform attribute of media object element
    InlineTransformations Module
    apply transformationFilter element directly to Media Object Element
  • transformation effect types
    • visualDeform -- mosaic, frosting and crystal effects
    • colorEffect -- monotone, sepia and negative
    • chromaKey -- blue-screen
    • soundEffect -- hall, theater and stadium
Affect:
SMIL 2.0 Modules (Addition)

Timed Text

Requirement:
Animated Subtitles
  • single standard
    • Title, Subtitles, Captions, Baloons
Proposal:
  • W3C Timed Text Working Group will work on this !

Profiles, Conformance

Requirement:
lightweight profile and conformance
Proposal:
  • slim-down of minimum profile
    • SMIL 2.0 Basic Profile requires 11(10) modules
      • Structure Module
      • Media Object Modules -- BasicMedia
      • Layout Modules -- BasicLayout
      • Timing and Synchronization Modules -- BasicInlineTiming, MinMaxTiming, BasicTimeContainers, RepeatTiming, (SyncbaseTiming)
      • Content Control Modules -- BasicContentControl, SkipContentControl
      • Linking Modules -- BasicLinking
    • all is not necessary and too many from the professional content authoring viewpoint
    • Tiny Profile ? Core Profile ?
    • needs more consideration

Summary

Thank you !