- From: Francois Daoust <fd@w3.org>
- Date: Thu, 09 Dec 2010 17:42:07 +0100
- To: Rich Tibbett <rich.tibbett@gmail.com>
- CC: public-device-apis@w3.org
Hi Rich, First, please note that the use cases I described were meant for a Media Gallery API in general, not as use cases for the existing Media Gallery API draft in particular. Next, the thing is: I don't think we disagree, actually. In particular, I agree that access to file metadata should be built on top of the FileSystem API. Whether that extension needs to be defined in the FileSystem API or in a separate Gallery API is a question of people, timing and priorities. On top of access to file metadata, what a Media Gallery API could bring is a performant and scalable interface to interact with the user's media gallery. The ability to mount folders is certainly a good idea worth pushing for, but I don't think it's enough to address all the use cases, IMO: - if the user accesses 100 Web sites that need to index a large portion of the user's media library, 100 different (sets of) indexes need to be computed, on the client. That's processor intensive. For instance, on my desktop computer, music indexation takes about 10mn (granted, it does that in the background, without eating too much CPU) and generates several Mb of database-like content. A Web site would need to spend that amount of time before it can offer methods to e.g. present content grouped by certain keywords/metadata. Or it would need to store several Mb of data in an offline storage, or to download it from the cloud. - synchronization between two different sessions would involve monitoring files in the folder (based on last modification date or a checksum?) to add/update/remove content in the database. Again, that would have to be done by each and every Web site that needs the info and would take a bit of time. In both cases, I think the problem is that the different indexes should rather be created and maintained by the OS or by the browser itself, and not on a case by case basis. Having the FileSystem API as the only means to interact with the media gallery would not allow that. Constrained devices would benefit from a Media Gallery API, especially because one can already store some pretty big amount of media on mobile devices. Then (but I'm not sure this should be addressed at that stage), the Media Gallery API could provide a controlled and secure way to interact with the user's media gallery, as oppposed to the FileSystem API which, by definition, operates at the file level. That's the fine-grained permission settings you allude to, I think, and which I probably did not manage to explain properly in my first email. A few more specific comments below. On 12/08/2010 05:16 PM, Rich Tibbett wrote: > Hi Francois, > > Thanks for taking the time to provide these use cases. > > I see all of these use cases being satisfied with a few additions to the > FileSystem API, rather than creating a Gallery API. Because Gallery > Objects are simply certain types of files it would be better if this > could build on top of the FileSystem API. > > I'm therefore still not convinced that we need a Gallery API. What I'd > like to propose is that you consider what requirements would be needed > on the File System API itself based around this same set of use cases > (e.g. the ability to mount folders in the FileSystem API). > > We may need meta data relating to media files but if the File object > could be capable of returning this information, perhaps we can cover > most of these use cases within a much more generic API. > > More specific feedback per use case below. > > - Rich > > On Fri, Dec 3, 2010 at 4:27 PM, Francois Daoust <fd@w3.org > <mailto:fd@w3.org>> wrote: > > Hello DAP working group, > > Dom took ACTION-296 during DAP F2F in Lyon to provide use cases for > the media gallery API. Dom and I have been exchanging ideas about > that, which lead to the following list of use cases, starting with > the ones already listed in the gallery draft: > http://dev.w3.org/2009/dap/gallery/#use-cases-and-requirements > > For each use case, I've detailed a short user story, highlighted > what would be needed, and added one or more comments. > > HTH, > Francois. > > > ========= > Use cases > ========= > > > Similar content recommendation > ----- > User story > -- > See use case 1 in gallery draft: > http://dev.w3.org/2009/dap/gallery/#uc1 > > What's needed > -- > - Media gallery API used as a way to filter out the user's > collection based on specific terms > > Comment > -- > While doable at a lower level through the FileSystem API, having to > parse thousands of pictures metadata to filter out the ones that > match specific terms is likely to take way too much time for > something as simple as a picture submission page. The possibility to > maintain an offline local database to speed up further submissions > would also be an overkill: there should be only one such database, > maintained at the browser (or OS) level, not one per Web site that > needs the info. > > > Any advances on this would be best left to the OS level and their file > pickers. For example, the Mac OS X file picker today lets me filter by > file name or contents (which could in the future e.g. also parse over > EXIF data or other meta data types). > > I don't think the provided use case is a particularly good one. Users > today still manage to upload their billions of photos successfully to > Facebook without this feature highlighting which photos it thinks > (perhaps incorrectly) that you want to upload. True. I agree. > Access to user categorization > ----- > User story > -- > See use case 2 in gallery draft: > http://dev.w3.org/2009/dap/gallery/#uc2 > (note the use case can be directly adapted to playlists for songs) > > What's needed > -- > - Access to user-generated metadata to filter out content > > > This could be achieved with the FileSystem API and extending the > returned File objects to also provide meta content for specific media > file types (audio/video/images). I guess that's precisely what a Media Gallery API would provide: a MediaMetadata interface, that could derive from the Metadata interface defined in the FileSystem API. There is also a clear overlap with the API for Media Resource 1.0. In the end, it has to be defined "somewhere", with a useful set of properties that make sense to address common needs (I mentioned the "artwork" which is not defined in the ontology as an example). > Music player > ----- > User story > -- > Clementine goes to http://example.com/musicplayer and grants the Web > site access to her local music repository. The music player lets her > browse her library, visualize metadata available (artworks, authors, > interpreters), and play songs. > > > If this is going to interact with the local device, perhaps the > FileSystem API is the solution (as above): mount your music folders to a > file system accessible by the web site and let it read and write from/to > there. Right. I think the problem is scaling. A full music player is perhaps not a good use case, since users tend to have only one music player. Think of different Web sites that propose different kinds of music-related services: - a Web site where you can browse your music collection and find corresponding lyrics - a Web site where you can see comments from others on music you have - a shopping Web site that tells you about new albums from artists you already own. These services are traditionally proposed as "add-ons" to music players, the media gallery API would allow them to be offered as separate Web sites, without having to duplicate the user's media library database (the indexes, that is). > What's needed > -- > - The music player needs access to specific content metadata > - Metadata should include author, actor, song type, as well as > lyrics or artwork. > - Access to metadata should be high-level. For instance, the artwork > of an album is basic stuff and would rather be expose in an > "artwork" property, but the Ontology for Media Resources seems to > leave it to the generic "ma:relation" property: > http://www.w3.org/TR/mediaont-10/#core-property-definitions > > > The FileSystem API should provide file meta data when returning File > objects. It should certainly build on top of the FileSystem API. That does not necessarily mean it needs to be defined in the same spec. I'm fine either way. > Comment > -- > The use case is obvious for the media gallery API. It will help > identify a common list of properties that need to be exposed through > the API. > > > > Music manager > ----- > User story > -- > Clementine goes to http://example.com/musicmanager and grants the > Web site access to her local music repository. The music manager > includes a music player as above. Additionally, it lets her manage > playlists, maintain her library (add/remove songs, categorize, edit > metadata, etc), and sync up with a portable music player with > limited storage. > > What's needed > -- > - same as above for music player part. > - the addition of a "manager" side means a fine-grained access to > metadata. While the player could make do with basic properties, a > manager will need more precise information. For instance, the > possibility to sync up with an external portable music aplayer > device probably means that the music player needs to have access to > the underlying file size and its encoding parameters. > - ideally, the music manager should also be able to access and > maintain statistics (usage statistics) or user ranking to organize > the user collection. > - read/write rights are needed, but these rights should be > restricted to the media gallery, and can be further restricted (e.g. > the application could only be allowed to manage playlists and not be > allowed to update content metadata). > > > I don't see why read/write needs to be restricted to the media gallery. > A web site can still store non-media stuff in your media directory > regardless of if you restrict access only to that point. If you allow a > web site to read/write to your disk then you cannot place restrictions > on what they can and cannot do in that space without more fine-grained > permissioning. > > We're trying to design out fine-grained permissioning points such as this. I didn't mean to say that the write rights would allow the manager to write any chunk of data to some portion of the local hard drive. I was actually thinking about a more fine-grained API, e.g. an "addMediaFileToPlaylist(DOMString uri)" method to manage songs in a playlist. The browser could then check that the URI actually refers to an existing media file and refuse to save data if that's not the case. Something similar could be done with "setters" for metadata properties. The goal being precisely not to allow the Web page to write directly to the hard drive. The Gallery API would thus provide a controlled way to interact with the user's media gallery. > Comment > -- > The use case is the most complete use case for the media gallery API. > It is the most fertile use case for the API since it will push for > the inclusion of more properties that the manager can take advantage of. > While read/write rights are needed to create a complete music > manager application, a music manager may still provide useful > features with very limited rights, possibly restricted to access to > an offline storage (although this would mean the information cannot > be exposed back to the media gallery API and "shared" with other > application or Web sites). > > > The File System API will allow you to mount your music folder(s) and/or > any USB music player(s) and it will manage your music collection and the > flow of files to/from those different mount points via that API. > > > Cloud-based gallery > ----- > User story > -- > Clementine manages her pictures collection online on > http://example.com/galleriedesglaces/ and wants to let > http://example.com/blah access this collection in the context of > e.g. the "Access to user categorization" use case already mentioned. > > What's needed > -- > - The media gallery API as a whole! > > Comment > -- > This use case highlights the need for a media gallery API as a > whole: while other use cases could be emulated using the FileSystem > API, it is quite unlikely that online services will expose something > similar to the FileSystem API. They may expose a more focused media > gallery API to let users access and manage their media content. > > > In short, the media gallery API can easily be mapped to a > cloud-based media gallery, while the FileSystem API is restricted to > "local" storage (where "local" may include remote storage, but not > online services that provide some other specific functionality). > > > I'm not sure I follow. If website1 wants to share data with website2 > then website1 should define an API for website2 to interact with. That's > a different use case to interfacing between the device and the web. > > It needs an API (and perhaps a standard one) but has different > requirements to the device<>web model. Use case is out of scope. Point taken. Targeting different models is probably not a good idea to sort this out. > Multimedia shows > ----- > User story 1 > -- > Ten years after the end of her studies, Clementine invites her old > schoolmates to meet for a "good old times" party at home. She has a > collection of pictures from that period, tagged with appropriate > metadata (caption, people, annotations, places) and would like to > run some animation on her connected TV in the background. She > browses http://example.com/slideshow and points it out to its > collection of pictures. She also makes sure the Web site has access > to her music repository. > > The Web site uses the media gallery API to extract relevant metadata > from the pictures, and order the slideshow consequently. It also > searches through Clementine's music collections for music that would > best match the pictures. It creates a personalized slideshow out of it. > > > Clementine can mount her connected TV to the web site/page via the > FileSystem API. Web site will then push the content that Clementine > chooses to that file system for creating the slideshow (or the website > creates a slideshow and pushes the completed file to the connected TV > file system). She could, yes. As mentioned at the beginning of this email, building the indexes needed before the site can start playing content is processor intensive. The Media Gallery API would avoid that step and provide an interface to a unique indexed database. > User story 2 > -- > Clementine is a 4-year-old little girl who wants to hear a great > story before she goes to bed. Her father uses a tablet and browses > http://example.com/childrenstories/ > The Web site proposes to personalize stories and asks Clementine's > father for a few keywords (name of an animal, name of the child, a > place or address, etc) > The Web site personalizes the story based on the father's responses > and adds illustrations taken from the pictures gallery that match > these responses as well as keywords from the story itself. > > What's needed > -- > - Need to be able to search for specific terms in > pictures/songs/movies metadata. > - Searchable metadata should include author, actor, song type, more > generic metadata (e.g. persons that appear on a picture), history of > a picture (time and location), as well as lyrics (or "subtitles" for > movies). > > Comment > -- > Animoto is a good example of what the service could look like: > http://animoto.com/ > The media gallery API would make it easier to do it using regular > Web technologies. > > > This content could have been mounted with the FileSystem API. The > website could pre-index the contents of that mount point and extract all > the content (file names + meta data) and store in the cloud. Then in > real-time as the user interacts with the web page, it can check this > index and go off and retrieve the relevant files from the local device > for display as required. True, but that means uploading several Mb of data before you can actually do something useful. It just seems too heavy. > Recommendation engine > ----- > User story > -- > Clementine wants to buy an electronic book on > http://example.com/ebooks but does not know what to select in the > myriad of books available. The Web sites browses her electronic > books collection and highlights a selection of books that have > something in common with the ones she already has in her collection > (same authors, similar theme, same edition, etc). > > What's needed > -- > - Same as above: some ability to search for specific terms in books > collection. > > > Again, assuming the website has access to my 'books' directory, it could > pre-index those files and their meta data and produce recommendations > from that. And again, I don't disagree ;) > Comment > -- > The user's local books collection may not adequately represent her > taste, but ranking could be taken into account by some other means > (left to the search engine) or exposed through the API. > > The media gallery service enables lightweight suggestion mechanisms > and personalization when a service does not know anything about the > user. > > > Personalized user experience > ----- > User story > -- > Clementine goes to online magazine http://example.com/magazine/ > There she reads short novels, articles on famous artists, fashion, > sports, culture, cinema, etc. > > The Web site provides contextual information to Clementine's music > player through the media gallery API. This contextual information is > based on keywords associated with the page the user is currently > looking at. Clementine's music player reacts to the contextual > information by selecting next song based on the provided context. > Thus Clementine gets to read articles while listening to contextual > music taken out from her own collection. > > What's needed > -- > - Need to be able to provide contextual information: keywords and/or > possibly qualified information such an author, an artist, a place, a > time, ... > > Comment > -- > In the other use cases, the media gallery API exposes information to > a Web page. This use case highlights how the media gallery API > could, in turn, let a Web page expose metadata to the user's media > gallery. > > The main benefit of this approach is that it makes it possible and > easy for a Web page to provide a more immersive experience without > having to access privacy-sensitive information. > > Obviously, the information could be exposed by means of, say, RDFa, > without requiring an API. The API would simply make the binding more > explicit. > > > > Again, this is a case for being able to pre-index file meta data > information made available when I have mounted a specific media folder > and made that accessible to the current web site. In that case, allowing the Web page to gain access to the whole user music gallery is too sensitive. There is no reason to let the newspaper know what the user gallery contains for the sake of a slightly more immersive experience. Still, this use case was more meant to convey how things could be seen from "another angle". I think it is out of scope of the Media Gallery. > ========== > Conclusion > ========== > > Main benefits of the media gallery API: > - high-level access to metadata (instead of having to ship hundreds > of Kb of JavaScript that handle things at a lower level), to allow > for the implementation of a complete music player and manager, and > to make cloud-based media galleriespossible. > - a standard and efficient search-oriented API to extract content > from user's media collection based on specific terms or metadata > - sandboxed write access to media gallery. Write access may not be a > requirement to start with. Most use cases and functionalities can be > implemented without it. > > > I'd actually prefer that we focus these use cases around the FileSystem > API. It would be good if within the FileSystem API I were able to mount > specific folders within my filesystem and then grant access to those > folders at runtime. Currently the FileSystem API is based on the ability > to provide access to a sand-boxed temporary/permanent file system > environment instead of being able to mount or symlink existing folders > and file systems. I think the main point is to give access to an indexed database that is maintained under the hood, and avoid the need for any application to do that on its own. > It also helps to take a more generic approach and build such use cases > on file systems in general. I'm fine if you want to map these use cases to file systems. After all, an index on artist and albums is a tree, and a tree can easily be mapped to a file system through symlinks. In short, you could "mount" different views as folders/files, but there would still need to be a need for some naming convention for the application to know what the view represents (to search for specific albums, for instance). The final solution is really up to to you, guys ;) Francois.
Received on Thursday, 9 December 2010 16:42:37 UTC