Use cases and requirements for Media Fragments

4.2.1.1 Scenario 1: Search Engine

Tim does a keyword search on a video search service. That keyword is found in several videos in the search service's collection and it relates to clips inside the videos that appear at a time offset. Tim would like the search result to point him to just these media fragments so he can watch the relevant clips rather than having to watch the full videos and manually search for the relevant clips.

4.2.1.2 Scenario 2: Region of an Image

Tim has discovered on an image hosting service a photo of his third school year class. He is keen to put a link to his own face inside this photo onto his private Web site where he is collecting old photos of himself. He does not want the full photo to be displayed and he does not want to have to download and crop the original image since he wants to reference the original resource.

4.2.1.3 Scenario 3: Portion of Music

Tim is a Last.fm user. He wants his friend Sue to listen to a cool song, Gypsy Davy. However, not really the entire song is worth it, Tim thinks. He wants Sue to listen to the last 10 seconds only and sends her an email with a link to just that subpart of the media resource.

4.2.1.4 Scenario 4: Moving Windows of Interest

Tim is now creating an analysis of the movements of muscles of horses during trotting and finds a few relevant videos online. His analysis is collected on a Web page and he'd like to reference the relevant video sections, cropped both in time and space to focus his viewers' attention on specific areas of interest that he'd like to point out.

4.2.2.1 Scenario 1: Segmenting a Video

Michael has a Website that collects recordings of the sittings of his government's parliament. These recordings tend to be very long - generally on the order of 7 hours in duration. Instead of splitting up the recordings into short files by manual inspection of the change of topics or some other segmentation approach, he prefers to provide many handles to a unique video resource. As he publishes the files however, he provides pagination on the videos such that people can watch them 20 min at a time.

4.2.2.2 Scenario 2: Temporal Audio Pagination

Lena would like to browse the descriptive audio tracks of a video like she does with Daisy audio books, by following the logical structure of the media. Audio descriptions and captions generally come in blocks either timed or separated by silences. Chapter by chapter and then section by section she eventually jumps to a specific paragraph and down to the sentence level by using the "tab" control like she would normally do in audio books. The descriptive audio track is an extra spoken track that provides a description of scenes happening in a video. When the descriptive audio track is not present, Lena can similarly browse through captions and descriptive text tracks which are either rendered through her braille reading device or through her text-to-speech engine.

4.2.2.3 Scenario 4: Spatial Video Pagination

Elaine has recorded a video mozaic of all her TV channels of an international election day. She wants to keep the original file of what all TV broadcasts are synchronously showing but now she wants to be able to make a long presentation where each channel will show, one at a time, one after another. She creates a playlist of media fragments URIs that each select a specific channel in the mozaic to play each channel after one another.

4.2.2.4 Scenario 5: Audio Passage Bookmark

Sue likes the song segment that Tim has sent her and decides to add this specific segment to her bookmarks.

When regarding media resources (in particular audio and video) as monolithic blocks, they are very inaccessible. For example, it is difficult to find out what they are about, where the highlights, or what the logical structure of the resources are. Lack of these features, in particular lack of captions and audio annotations, further make the resources inaccessible to disabled people. Introducing an ability to directly access highlights, fragments, or the logical structure of a media resource will provide a big contribution towards making a media resource more accessible.

4.2.2.5 Scenario 6: Captions helps browsing Video

Silvia has a deaf friend, Elaine, who would like to watch the holiday videos that Silvia is publishing on her website. Silvia has created subtitle tracks for her videos and also a CMML annotation with unique identifiers on the clips that she describes. The clips were formed based on locations that Silvia has visited. In this way, Elaine is able to watch the videos by going through the clips and reading the subtitles for those clips that she is interested in. She watches the sections on Korea, Australia, and France, but jumps over the ones of Great Britain and Holland.

4.2.3.1 Scenario 1: Reframing a photo in a slideshow

Erik has a collection of photos and wants to create a slide show of some of the photos and wants to highlight specific areas in each video. He uses xspf to define the slide show (playlist) using spatial fragment URIs to address the photo fragments.

4.2.3.2 Scenario 2: Mozaic

Jack wants to create a mosaic for his website with all the image fragments that Erik defined collated together. He uses SMIL 3.0 Tiny Profile and the spatial fragment URIs to layout the image fragments and stitch them together as a new "image".

4.2.3.3 Scenario 3: Video Mashup

Jack has a collection of videos and wants to create a mashup from segments out of these videos without having to manually edit them together. He uses SMIL 3.0 Tiny Profile and temporal fragment URIs to address the clips out of the videos and sequence them together.

Given an ability to link to media fragments through URIs, people will want to determine whether they receive the full resource or just the data that relates to the media fragment. This is particularly the case where the resource is large, where the bandwidth is scarce or expensive, and/or where people have limited time/patience to wait until the full resource is loaded.

4.2.3.4 Scenario 4: Selective previews

Yves is a busy person. He doesn't have time to attend all meetings that he is supposed to attend. He also uses his mobile device for accessing Web resources while traveling, to make the most of his time. Some of the recent meetings that Yves was supposed to attend have been recorded and published on the Web. A colleague points out to Yves in an email which sections of the meetings he should watch. While on his next trip, Yves goes back to this email and watches the highlighted sections by simply clicking on them. The media server of his company dynamically composes a valid media resource from the URIs that Yves is sending it such that Yves' video player can play just the right fragments.

4.2.3.5 Scenario 5: Music Samples

Erik also has a music collection. He creates an "audio podcast" in the form of an RSS feed with URIs that link to samples from his music files. His friends can play back the samples in their Web-attached music players.

4.2.3.6 Scenario 6: Highlighting regions (Out-Of-Scope)

Tim has discovered yet another alumni photo of his third school year class. This time he doesn't want to crop his face but he wants to keep the photo in the context of his classmates. He wants his region of the photo highlighted and the rest grey scaled.

4.2.4.1 Scenario 1:Spatial Tagging of Images

Raphael systematically annotates some highlighted regions in his photos that depicts his friends, families, or the monuments he finds impressive. This knowledge is represented by RDF descriptions that use spatial fragment URIs to relate to the image fragments in his annotated collection. It makes it possible later to search and retrieve all these media fragment URIs that relate to one particular friend or monument.

4.2.4.2 Scenario 2: Temporal Tagging of Audio and Video

Raphael also has a collection of audio and video files of all the presentations he ever made. His RDF description collection extends to describing all the segments where he gave a demo of a software system with structured details on the demo.

NB: Time-aligned text such as captions, subtitles in multiple languages, and audio descriptions for audio and video don't have to be created as separate documents and link to each segment through a temporal URI. Such text can be made part of the media resource by the media author or delivered as a separate, but synchronised data stream to the media player. In either case, it should be made accessible in a Web page through a javascript API or access through a DOM nested browsing context of the video/audio/image element. This needs to be addressed in the HTML5 working group.

Annotating media resources on the level of a complete resource is in certain circumstances not enough. Support for annotating multimedia on the level of fragments is often desired. The definition of "anchors" (or id tags) for fragments of media resources will allow us to identify fragments by name. It allows the creation of an author-defined segmentation of the resource - an author-provided structure.

4.2.4.3 Scenario 3: Named Anchors

Raphael would like to attach an RDF-based annotation to a video fragment that is specified through an "anchor". Identifying the media fragment by name instead of through a temporal video fragment URI allows him create a more memorable URI than having to remember the time offsets.

4.2.4.4 Scenario 4: Spatial and Temporal Tagging

Guillaume uses video fragments URIs in an [MPEG-7] sign language profile to describe a moving point of interest: he wants the focus region to be the dominant hand of in a Sign Language video. Not only the series of video fragment URIs gives the coordinates and timing of the trajectory followed by the hand, it can also describe the areas of changing handshapes.

4.2.4.5 Scenario 5: Search Engine

Guillaume wants to retrieve the images of each bike present at a recent cycling event. Group photos and general shots of the event have been published online and thanks to a query in a search engine, Guillaume can now retrieve multiple individual shots of each bike in the collection.

4.2.5.1 Scenario 1: Changing Video quality (Out-Of-Scope)

Davy is looking for videos about allergies and would like to get previews at a lower frame rate to decide whether to download and save them in his collection. He would like to be able to specify in the URI a means of telling the media server the adaptation that he is after. For video he would like to adapt width, height, frame rate, colour depth, and temporal subpart selection. Alternatively, he may want to get just a thumbnail of the video.

Note: This scenario is out of scope for this Working Group because it requires changes be made to the actual encoded data to retrieve a "fragment". URI-based media fragments should basically be achieved through cropping of one or more byte sections.

4.2.5.2 Scenario 2: Selecting Regions in Images

Davy is interested to have precise coordinates on his browser address bar to see and pan over large-size images maps. Through the same URI scheme he can now generically address and locate different image subparts on his client side for all image types.

4.2.5.3 Scenario 3: Selecting an Image from a multi-part document

Davy is now interested in multi-resolution, multi-page medical images. He wants to select the detailed image of the toe X-rays which appear on page 7 of the TIFF document.

4.2.5.4 Scenario 4: Retrieving an Image embedded thumbnail

Davy is also interested to have the kind of preview functionality for pictures, in particular these JPEG large 10 Mega pixels files that have embedded thumbnails in them. He can now provide a fast preview by selecting the embedded thumbnail in the original image without even having to resize or create a new separate file!

4.2.5.5 Scenario 5: Switching of Video Transmission

Davy has a blind friend called Katrina. Katrina would also like to watch the videos that Davy has found, and is lucky that the videos have additional alternative audio tracks, which describe to blind users what is happening in the videos. Her Internet connection is of lower bandwidth and she would like to switch off the video track, but receive the two audio tracks (original audio plus audio annotations). She would like to do this track selection through simple changes to the URI.

4.2.5.6 Scenario 6: Toggle All Audio OFF

Sebo is Deaf and enjoys watching videos on the Web. Her friend sent her a link to a new music video URI but she doesn't want to waste time and bandwidth receiving any sounds. So when she enters the URI in her browser's address bar, she also adds an extra parameter to ignore all audio tracks without naming them by selecting the video segment only.

4.2.5.7 Scenario 7: Toggle specific Audio tracks

Davy's girlfriend is a fan of Karaoke. She would love to be able to play back videos from the Web that have a karaoke text, and two audio tracks, one each for the music and for the singer. Then she could practice the songs by playing back the complete video with all tracks, but use the video in Karaoke parties with friends where she turns off the singer's track through a simple selection of tracks in the User Agent.