Re: Streaming with Media Source

Hi Aymeric,

Commetns inline.


On Wed, Mar 26, 2014 at 10:38 AM, Aymeric Vitte <vitteaymeric@gmail.com>wrote:

>  Thanks for your answers.
>
> Submitting the chunks of a fragmented mp4 file (created with MP4Box for
> example) does not work in Chrome, as far as I understand this is not normal.
>

This should work. It is hard to help you figure out what is wrong w/o a
specific example and this mailing list is probably not an appropriate place
for UA specific support. I assure you that many people have created
fragmented MP4 with MP4Box and it works fine with Chrome's MSE
implementation. Common mistakes are not specifying the codecs that are
actually used by the file and various issues related to how they are
appending data. Like I've said in earlier emails in this thread,
chrome:media-internals is helpful for uncovering other common issues people
encounter with files.


>
> If you are including MP3, can't you include non fragmented MP4 and WebM?
>

- We've added support to allow the MSE implementation to accept a stream of
MP3 frames. This is slightly different than handling an MP3 file since it
is up to the web application to properly figure out where seek points in an
MP3 file are. MP3 frames are essentially tiny media fragments and don't
have any file offsets to worry about so it is pretty straightforward to
support in the MSE context.
- Handling non-fragmented MP4 is non-trivial especially when it contains
absolute file offsets in it. It also is very loose about box ordering so a
simple byte stream abstraction, like what MSE provides, is not sufficent.
It is not what MSE was designed to do.
- WebM by default is essentially fragmented. Each cluster in a WebM file is
self contained and is roughly equivalent to an (moof,mdat) pair in an MP4
fragment. The only elements in a WebM file that contain absolute offsets
are cues & the metaseek header and those are explictly ignored by MSE.
Given this, it is up to the web application to figure out the size and
location of clusters which is usually where some form of manifest comes
into play.


>
> How can I achieve what I want to do? This would be incredible that this is
> not feasible.
>

If you transmux your MP4 into fragmented MP4 you should be able to do what
you want. I'm aware of several commercial entities that have successfully
done P2P distribution of fragmented MP4 content w/ MSE & WebRTC.

If you just want to send raw chunks of arbitrarily formated media files,
then you are out of luck right now. I'm not aware of any web platform API
that will let you do this.

Aaron


>
> Regards
>
> Aymeric
>
> Le 26/03/2014 17:32, Aaron Colwell a écrit :
>
> Hi Aymeric,
>
> Comments inline...
>
> On Wed, Mar 26, 2014 at 8:24 AM, Aymeric Vitte <vitteaymeric@gmail.com>wrote:
>
>> Thanks, yes we can share experiences (I thought it was not implemented in
>> FF, from which version?), I saw your tutorial already, very well explained,
>> what I want to do is a bit different and very simple:
>>
>> Usually you use something like <video src='movie.mp4' />, then you get
>> the chunks via HTTP and the video is displayed.
>>
>> I want to do exactly the same thing, I receive the chunks (from the
>> Peersm protocol via WebSockets or WebRTC Data Channels) and MSE displays
>> the movie.
>>
>
>  MSE was not designed for this. To support this in the generic case the
> code fetching the data needs to understand the file format so it can
> properly fetch the necessary chunks for playback and make tradeoffs about
> the size of the chunk to fetch vs the amount of bytes actually needed in
> that chunk.
>
>
>>
>> I have read from some Aaron posts that MSE only works with fragmented
>> files, in addition you have to submit the fragments as defined in the
>> manifest file (or at least the initialization segment? I have not tested
>> this for now).
>>
>
>  The manifest file isn't required by MSE, but that is typically how specs
> like DASH describe the layout & location of the fragments.
>
>
>>
>> It does not work for mp3 (as far as I saw the Web Audio API does not
>> allow to do what I want to do too).
>>
>
>  I plan on submitting an MSE byte stream format specification for MP3 and
> AAC in ADTS in the next day or two. I'm just putting the final touches on
> the language. Chrome has experimental support for both of these formats
> right now.
>
>
>>
>> Format available inside the browsers are not numerous, now the API is
>> rectricting to fragmented files, so I find it extremely limited.
>>
>
>  I'm sorry you feel this way. Fragmented files allow segments of media to
> be self contained and only contain local references. This makes dealing
> with chunked media much easier. For the generic case you have to deal with
> absolute file references and arbitrary file layout which is much more
> difficult to work with. The API would look very different and would not be
> able to provide many of the features that MSE does.
>
>
>>
>> I have some hard time believing that no API did envision that you might
>> want to stream video and audio based on the received chunks, is it the case?
>>
>
>  Again, your use case is different from what MSE goals were. It appears
> that you simply want to replace the UAs HTTP fetching code with your own
> P2P code. That is fine, but not what MSE was designed to solve.
>
>
>>
>> Do IE and FF only implement it too for fragmented files?
>>
>
>  Yes. They conform to the MSE spec<https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/media-source.html>and implement the requirements laid out in the byte
> stream format specifications<https://dvcs.w3.org/hg/html-media/raw-file/default/media-source/byte-stream-format-registry.html> they
> have chosen to support. For MP4 in particular, this means fragmented files
> only.
>
>  I hope this helps you understand,
>
>  Aaron
>
>
>>
>> Regards
>>
>> Aymeric
>>
>> Le 25/03/2014 17:46, Jay Munro a écrit :
>>
>>  How are you preparing your source files? Do they have the initialization
>>> box/packet (normally on any mp4 or webm file) for at least the first
>>> segment?
>>>
>>> I've created a MSE player for DASH mp4 files.. I'm not an expert, but
>>> I've gotten it to work on a single stream on IE and Firefox. I would like
>>> to get it to work on chrome, so maybe we can share experiences.
>>>
>>> I don't know if this will help, but here's a blog post I wrote on one
>>> way to do it (there seems to be quite a few)
>>> http://blogs.msdn.com/b/mediastuff/archive/2014/02/19/video-streaming-without-plug-ins-or-special-servers.aspx
>>>
>>> -Jay
>>>
>>>
>>> -----Original Message-----
>>> From: Aymeric Vitte [mailto:vitteaymeric@gmail.com]
>>> Sent: Tuesday, March 25, 2014 4:23 AM
>>> To: public-html-media@w3.org
>>> Cc: acolwell@chromium.org
>>> Subject: Streaming with Media Source
>>>
>>> Hi,
>>>
>>> For [1] I have started to implement Streaming for audio and video using
>>> the Media Source API, the app is retrieving chunks and is appending them to
>>> the "source" of audio/video tags.
>>>
>>> This was supposed to be simple but it is not, the API (or at least the
>>> Chrome implementation) seems to handle only adaptive rate/manifest
>>> structures both for mp4 and webm, it's unclear if the audio tag is
>>> supported.
>>>
>>> I am not an expert in video/audio format but I don't see very well
>>> what's the issue to append chunks to a destination, or to turn it into a
>>> stream (since I am participating to the Streams API, I know it's not there
>>> yet) outside of other interfaces such as WebRTC.
>>>
>>> For the final phase of the project, WebRTC will be used but with
>>> DataChannels, so the issue remains.
>>>
>>> I have looked in all existing APIs and unless I am missing something, I
>>> still don't see how to achieve this simple thing, how can it be done?
>>>
>>> Regards,
>>>
>>> Aymeric
>>>
>>> [1] http://www.peersm.com
>>>
>>> --
>>> Peersm : http://www.peersm.com
>>> node-Tor : https://www.github.com/Ayms/node-Tor
>>> GitHub : https://www.github.com/Ayms
>>>
>>>
>>>
>> --
>> Peersm : http://www.peersm.com
>> node-Tor : https://www.github.com/Ayms/node-Tor
>> GitHub : https://www.github.com/Ayms
>>
>>
>
> --
> Peersm : http://www.peersm.com
> node-Tor : https://www.github.com/Ayms/node-Tor
> GitHub : https://www.github.com/Ayms
>
>