Re: MSE byte-stream format initialization segment boxes

Hi David,

Comments inline.


On Wed, Feb 19, 2014 at 11:41 AM, David Singer <singer@apple.com> wrote:

> Hi, thanks, inline
>
> On Feb 18, 2014, at 13:13 , Aaron Colwell <acolwell@google.com> wrote:
>
> >
> >
> > On Tue, Feb 18, 2014 at 9:54 AM, David Singer <singer@apple.com> wrote:
> > Hi guys
> >
> > there is a sentence in
> >
> >
> http://www.w3.org/2013/12/byte-stream-format-registry/isobmff-byte-stream-format.html#iso-init-segments
> >
> > which is causing us some problems, notably the ‘ignore' here:
> >
> > > Valid top-level boxes such as ftyp, styp, and sidx are allowed to
> appear before the moov box.
> > > These boxes must be accepted and ignored by the user agent and are not
> considered part of the initialization segment in this specification.
> >
> > This is causing some implementations to strip these boxes before they
> get to the media engine, and then we’ve lost important compatibility
> information (notably the claims of compatibility made by the ftyp and styp
> boxes) and, if we want to index, the indexing information.  I suppose they
> think we’re going to conform to the apparent requirement (though it’s
> expressed as a statement of fact, rather than as an option or requirement)
> to ignore.  But why?
> >
> > How would the styp & ftyp boxes change how the byte streams are handled?
> They don't appear to provide much value in the MSE byte stream context.
>
> They provide a lot of value to the media engine;  pre-flighting whether or
> how to play, for example.  In one example, we require ftyp boxes on both
> movie (QT) files and MP4 files, and the engine uses the ftyp box to handle
> a few places where these must be interpreted differently.  Even without
> this case, it really helps to get early warning of possible problems.  Yes,
> the DASH MPD manifest should have quoted the file-type brands in the
> profiles parameter of the MIME type, so playability could be determined
> then, but if it didn’t, at least one gets a warning to the media engine
> before one encounters an unsupported feature.
>
>
After reading up on this a little more, I'm starting to come around to
making ftyp a required part of the initialization segment.


> > I don't think they should be required as part of the initialization
> segment especially if it introduced a restriction like, all ftyp/styp boxes
> must have the same major_brand, minor_version, and compatible_brands.
>
> They are required in both the ISO base media file format, and in the DASH
> specification, for good reasons.  I don’t think it’s the place of MSE to
> try to relax requirements in the specs it is building on.
>

I am starting to understand those reasons now. Please try to remember how
MSE started. I wanted to start with the simplest thing that could possibly
work. This meant only paying attention to the boxes that were absolutely
necessary and adding restrictions, like preventing usage of absolute
offsets, to make sure people didn't create insane content that didn't make
sense in the MSE context. ISOBMFF has so many features that I really wanted
to constrain it to a minimal useful set. I'm starting to see that an ftyp
brand is what I should have been using to "label" these groups of
constraints.

It would be really nice if you could provide some guidance on what brands
you intend to support. The rules in the spec right now appear to imply at
least 'iso6' and parts of 'dash'. It would probably be a good idea to
declare the minimal brand set for interop purposes. We may even want to
consider starting the definition of an MSE brand if an existing brand
doesn't sufficiently constain things. I have no idea what the appropriate
process is for that.


> > The sidx box contains file specific offset info. MSE does not require a
> full segment file to be appended so interpreting a sidx box is never
> guaranteed to be correct. MSE intentionally has no concept of file
> boundaries so, as far as I can tell, there is no way for an implementation
> to determine if the fragments that follow a sidx box are actually the ones
> the sidx box refers to. I intentionally wanted to break the "append a full
> file" requirement to provide maximum flexibility in presentation
> construction.
>
> OK, the segment index is something I don’t use.
>
> But the segment type box then is important; it allows you to notice that
> you have a segment in your hand that has differently compatibility claims
> than the initialization segment you are operating under.  Maybe something
> went wrong and you’re not in Kansas any more.
>

Fair enough. I just want to make sure that the spec doesn't explicitly
prevent changes in the brands. I mainly care about this because I want to
maximize the amount of content that can be spliced together with MSE. If a
UA supports brand X and brand Y, I'd like to allow that any combination of
brand X and brand Y can be used together. Obviously an initialization
segment and its associated media segments need to have compatible brands.


>
> > Should this be re-phrased?
> >
> > These boxes *are* considered part of the initialization segment in this
> specification and must be accepted and passed by the user-agent to the
> media engine; they may be ignored or processed as desired.
> >
> > I don't think we should use this wording for the reasons given above. In
> my opinion we should definitely ignore any top-level boxes that have file
> relative offsets in them or ones that make any assumptions about byte
> stream layout.
>
> OK, the ftyp/styp boxes have neither of these issues.
>
>
Agreed. I'll file a bug for making parsing & validating of ftyp/styp boxes
required.

Aaron