Re: Uses cases for the Media Gallery API

Hi Francois,

Thanks for taking the time to provide these use cases.

I see all of these use cases being satisfied with a few additions to the
FileSystem API, rather than creating a Gallery API. Because Gallery Objects
are simply certain types of files it would be better if this could build on
top of the FileSystem API.

I'm therefore still not convinced that we need a Gallery API. What I'd like
to propose is that you consider what requirements would be needed on the
File System API itself based around this same set of use cases (e.g. the
ability to mount folders in the FileSystem API).

We may need meta data relating to media files but if the File object could
be capable of returning this information, perhaps we can cover most of these
use cases within a much more generic API.

More specific feedback per use case below.

- Rich

On Fri, Dec 3, 2010 at 4:27 PM, Francois Daoust <fd@w3.org> wrote:

> Hello DAP working group,
>
> Dom took ACTION-296 during DAP F2F in Lyon to provide use cases for the
> media gallery API. Dom and I have been exchanging ideas about that, which
> lead to the following list of use cases, starting with the ones already
> listed in the gallery draft:
>  http://dev.w3.org/2009/dap/gallery/#use-cases-and-requirements
>
> For each use case, I've detailed a short user story, highlighted what would
> be needed, and added one or more comments.
>
> HTH,
> Francois.
>
>
> =========
> Use cases
> =========
>
>
> Similar content recommendation
> -----
> User story
> --
> See use case 1 in gallery draft:
> http://dev.w3.org/2009/dap/gallery/#uc1
>
> What's needed
> --
> - Media gallery API used as a way to filter out the user's collection based
> on specific terms
>
> Comment
> --
> While doable at a lower level through the FileSystem API, having to parse
> thousands of pictures metadata to filter out the ones that match specific
> terms is likely to take way too much time for something as simple as a
> picture submission page. The possibility to maintain an offline local
> database to speed up further submissions would also be an overkill: there
> should be only one such database, maintained at the browser (or OS) level,
> not one per Web site that needs the info.
>
>
Any advances on this would be best left to the OS level and their file
pickers. For example, the Mac OS X file picker today lets me filter by file
name or contents (which could in the future e.g. also parse over EXIF data
or other meta data types).

I don't think the provided use case is a particularly good one. Users today
still manage to upload their billions of photos successfully to Facebook
without this feature highlighting which photos it thinks (perhaps
incorrectly) that you want to upload.


>
>
> Access to user categorization
> -----
> User story
> --
> See use case 2 in gallery draft:
> http://dev.w3.org/2009/dap/gallery/#uc2
> (note the use case can be directly adapted to playlists for songs)
>
> What's needed
> --
> - Access to user-generated metadata to filter out content
>
>
This could be achieved with the FileSystem API and extending the returned
File objects to also provide meta content for specific media file types
(audio/video/images).


>
>
> Music player
> -----
> User story
> --
> Clementine goes to http://example.com/musicplayer and grants the Web site
> access to her local music repository. The music player lets her browse her
> library, visualize metadata available (artworks, authors, interpreters), and
> play songs.
>
>
If this is going to interact with the local device, perhaps the FileSystem
API is the solution (as above): mount your music folders to a file system
accessible by the web site and let it read and write from/to there.


> What's needed
> --
> - The music player needs access to specific content metadata
> - Metadata should include author, actor, song type, as well as lyrics or
> artwork.
> - Access to metadata should be high-level. For instance, the artwork of an
> album is basic stuff and would rather be expose in an "artwork" property,
> but the Ontology for Media Resources seems to leave it to the generic
> "ma:relation" property:
>   http://www.w3.org/TR/mediaont-10/#core-property-definitions
>
>
The FileSystem API should provide file meta data when returning File
objects.


> Comment
> --
> The use case is obvious for the media gallery API. It will help identify a
> common list of properties that need to be exposed through the API.
>
>
>
> Music manager
> -----
> User story
> --
> Clementine goes to http://example.com/musicmanager and grants the Web site
> access to her local music repository. The music manager includes a music
> player as above. Additionally, it lets her manage playlists, maintain her
> library (add/remove songs, categorize, edit metadata, etc), and sync up with
> a portable music player with limited storage.
>
> What's needed
> --
> - same as above for music player part.
> - the addition of a "manager" side means a fine-grained access to metadata.
> While the player could make do with basic properties, a manager will need
> more precise information. For instance, the possibility to sync up with an
> external portable music aplayer device probably means that the music player
> needs to have access to the underlying file size and its encoding
> parameters.
> - ideally, the music manager should also be able to access and maintain
> statistics (usage statistics) or user ranking to organize the user
> collection.
> - read/write rights are needed, but these rights should be restricted to
> the media gallery, and can be further restricted (e.g. the application could
> only be allowed to manage playlists and not be allowed to update content
> metadata).
>

I don't see why read/write needs to be restricted to the media gallery. A
web site can still store non-media stuff in your media directory regardless
of if you restrict access only to that point. If you allow a web site to
read/write to your disk then you cannot place restrictions on what they can
and cannot do in that space without more fine-grained permissioning.

We're trying to design out fine-grained permissioning points such as this.


>
> Comment
> --
> The use case is the most complete use case for the media gallery API.
> It is the most fertile use case for the API since it will push for the
> inclusion of more properties that the manager can take advantage of.
> While read/write rights are needed to create a complete music manager
> application, a music manager may still provide useful features with very
> limited rights, possibly restricted to access to an offline storage
> (although this would mean the information cannot be exposed back to the
> media gallery API and "shared" with other application or Web sites).
>
>
The File System API will allow you to mount your music folder(s) and/or any
USB music player(s) and it will manage your music collection and the flow of
files to/from those different mount points via that API.


>
> Cloud-based gallery
> -----
> User story
> --
> Clementine manages her pictures collection online on
> http://example.com/galleriedesglaces/ and wants to let
> http://example.com/blah access this collection in the context of e.g. the
> "Access to user categorization" use case already mentioned.
>
> What's needed
> --
> - The media gallery API as a whole!
>
> Comment
> --
> This use case highlights the need for a media gallery API as a whole: while
> other use cases could be emulated using the FileSystem API, it is quite
> unlikely that online services will expose something similar to the
> FileSystem API. They may expose a more focused media gallery API to let
> users access and manage their media content.


> In short, the media gallery API can easily be mapped to a cloud-based media
> gallery, while the FileSystem API is restricted to "local" storage (where
> "local" may include remote storage, but not online services that provide
> some other specific functionality).
>
>
I'm not sure I follow. If website1 wants to share data with website2 then
website1 should define an API for website2 to interact with. That's a
different use case to interfacing between the device and the web.

It needs an API (and perhaps a standard one) but has different requirements
to the device<>web model. Use case is out of scope.


>
>
> Multimedia shows
> -----
> User story 1
> --
> Ten years after the end of her studies, Clementine invites her old
> schoolmates to meet for a "good old times" party at home. She has a
> collection of pictures from that period, tagged with appropriate metadata
> (caption, people, annotations, places) and would like to run some animation
> on her connected TV in the background. She browses
> http://example.com/slideshow and points it out to its collection of
> pictures. She also makes sure the Web site has access to her music
> repository.
>
> The Web site uses the media gallery API to extract relevant metadata from
> the pictures, and order the slideshow consequently. It also searches through
> Clementine's music collections for music that would best match the pictures.
> It creates a personalized slideshow out of it.
>

Clementine can mount her connected TV to the web site/page via the
FileSystem API. Web site will then push the content that Clementine chooses
to that file system for creating the slideshow (or the website creates a
slideshow and pushes the completed file to the connected TV file system).


>
> User story 2
> --
> Clementine is a 4-year-old little girl who wants to hear a great story
> before she goes to bed. Her father uses a tablet and browses
> http://example.com/childrenstories/
> The Web site proposes to personalize stories and asks Clementine's father
> for a few keywords (name of an animal, name of the child, a place or
> address, etc)
> The Web site personalizes the story based on the father's responses and
> adds illustrations taken from the pictures gallery that match these
> responses as well as keywords from the story itself.
>
> What's needed
> --
> - Need to be able to search for specific terms in pictures/songs/movies
> metadata.
> - Searchable metadata should include author, actor, song type, more generic
> metadata (e.g. persons that appear on a picture), history of a picture (time
> and location), as well as lyrics (or "subtitles" for movies).
>
> Comment
> --
> Animoto is a good example of what the service could look like:
>  http://animoto.com/
> The media gallery API would make it easier to do it using regular Web
> technologies.
>

This content could have been mounted with the FileSystem API. The website
could pre-index the contents of that mount point and extract all the content
(file names + meta data) and store in the cloud. Then in real-time as the
user interacts with the web page, it can check this index and go off
and retrieve the relevant files from the local device for display as
required.


>
>
>
> Recommendation engine
> -----
> User story
> --
> Clementine wants to buy an electronic book on http://example.com/ebooksbut does not know what to select in the myriad of books available. The Web
> sites browses her electronic books collection and highlights a selection of
> books that have something in common with the ones she already has in her
> collection (same authors, similar theme, same edition, etc).
>
> What's needed
> --
> - Same as above: some ability to search for specific terms in books
> collection.
>

Again, assuming the website has access to my 'books' directory, it could
pre-index those files and their meta data and produce recommendations from
that.


> Comment
> --
> The user's local books collection may not adequately represent her taste,
> but ranking could be taken into account by some other means (left to the
> search engine) or exposed through the API.
>
> The media gallery service enables lightweight suggestion mechanisms and
> personalization when a service does not know anything about the user.
>
>
> Personalized user experience
> -----
> User story
> --
> Clementine goes to online magazine http://example.com/magazine/
> There she reads short novels, articles on famous artists, fashion, sports,
> culture, cinema, etc.
>
> The Web site provides contextual information to Clementine's music player
> through the media gallery API. This contextual information is based on
> keywords associated with the page the user is currently looking at.
> Clementine's music player reacts to the contextual information by selecting
> next song based on the provided context. Thus Clementine gets to read
> articles while listening to contextual music taken out from her own
> collection.
>
> What's needed
> --
> - Need to be able to provide contextual information: keywords and/or
> possibly qualified information such an author, an artist, a place, a time,
> ...
>
> Comment
> --
> In the other use cases, the media gallery API exposes information to a Web
> page. This use case highlights how the media gallery API could, in turn, let
> a Web page expose metadata to the user's media gallery.
>
> The main benefit of this approach is that it makes it possible and easy for
> a Web page to provide a more immersive experience without having to access
> privacy-sensitive information.
>
> Obviously, the information could be exposed by means of, say, RDFa, without
> requiring an API. The API would simply make the binding more explicit.
>
>
>
Again, this is a case for being able to pre-index file meta data information
made available when I have mounted a specific media folder and made that
accessible to the current web site.


>
> ==========
> Conclusion
> ==========
>
> Main benefits of the media gallery API:
> - high-level access to metadata (instead of having to ship hundreds of Kb
> of JavaScript that handle things at a lower level), to allow for the
> implementation of a complete music player and manager, and to make
> cloud-based media galleriespossible.
> - a standard and efficient search-oriented API to extract content from
> user's media collection based on specific terms or metadata
> - sandboxed write access to media gallery. Write access may not be a
> requirement to start with. Most use cases and functionalities can be
> implemented without it.
>
>
I'd actually prefer that we focus these use cases around the FileSystem API.
It would be good if within the FileSystem API I were able to mount specific
folders within my filesystem and then grant access to those folders at
runtime. Currently the FileSystem API is based on the ability to provide
access to a sand-boxed temporary/permanent file system environment instead
of being able to mount or symlink existing folders and file systems.

It also helps to take a more generic approach and build such use cases on
file systems in general.

- Rich