Re: Integer PCM sample formats to Web Audio API?

2014-01-10 18:27, Chris Wilson skrev:
> We don't actually even NEED to add anything to the API, we could 
> simply mention it's a possible optimization to implementations to 
> choose to keep integer-based representations in their original form 
> until getChannelData is called.
>
>

True, that will solve the specific case for emulators and similar 
applications that generate 16-bit data on the fly. However, it would not 
cover the case of saving memory for OGG-compressed samples and similar. 
You might want to tell the engine not to expand the data internally to 
Float32 (AFAIK it's not possible to tell whether an OGG/MP3/etc should 
be unpacked to 8/16-bit integers etc). OTOH one solution could be to 
allow decodeAudioData() to ALWAYS decode to Int16 (I think that very few 
samples would suffer any quality loss from such a limitation).

Another issue is the resampling step of decodeAudioData(). In order to 
save memory, you would not want a 22KHz OGG file to be resampled to 
48KHz, for instance (in combination with 16-bit -> 32-bit, that's >4x 
the size). Here, we could just change the wording in the spec and make 
the resampling step optional for the implementation.

Again, I'm not sure if we're better off with a developer-provided 
quality/savemem hint, or leaving it completely to the implementation.

Thoughts?


/Marcus



> On Fri, Jan 10, 2014 at 12:53 AM, Marcus Geelnard <mage@opera.com 
> <mailto:mage@opera.com>> wrote:
>
>     2014-01-09 20:49, Paul Adenot skrev:
>
>                    - A way to query whether a given buffer is int16 or
>                 float32, to allow
>                 code to assert() that it is getting what it desires.
>                 Otherwise debugging
>                 can be a pain if the only way is for the programmer to
>                 measure this in
>                 indirect ways via looking at memory consumption in a
>                 system process manager.
>
>             I would NOT want to do this.  This should be under the
>             hood, and some
>             systems (e.g. desktops with lots of memory) may choose to
>             always expand.
>
>         I'd rather do something explicit, yes, but see below.
>
>
>     I would like to avoid possibilities of inspecting engine internals
>     like this. It raises the risk of site breakage and limits the ways
>     in which an implementation can do optimizations.
>
>
>
>
>                    - If the support for int16 is specced in to be
>                 optionally supportable (I
>                 hope this will not be the case!), a way to ask the
>                 implementation if it
>                 does support int16 format.
>
>             I don't think you'd need this.  It's going to end up being
>             a performance
>             tradeoff - memory for CPU - that's likely better made by
>             the platform
>             implementation than the developer.  If we did want this,
>             I'd expect this is
>             a different decodeAudioData-style call.
>
>         I think that would address the use-case, here, while being
>         retro-compatible.
>
>         We could pass in an optional value as last argument of
>         decodeAudioData,
>         that would indicates that the author would rather save memory
>         than have
>         a good quality / save some CPU.
>
>
>     I agree with Chris here. In general, an implementation has a much
>     better chance of making an informed performance trade off decision
>     than a Web developer. In most situations I think that int16 would
>     save memory, but add some extra CPU overhead. In other words, if a
>     client has loads and loads of memory, it could for instance decide
>     to take the memory hit and convert to Float32 internally in order
>     to lower the CPU load, either up front or lazily.
>
>     If a Web page was to make the trade off decision, it would have to
>     know a lot more about the device it's running on (such as CPU
>     performance, available memory, etc), and since that is not the
>     case, I fear that Web devs would resort to things like UA sniffing
>     etc to make distinctions such as "desktop" vs "fast-tablet" vs
>     "slow-phone", etc.
>
>
>
>         It seems that it could be useful to specify sample rate and
>         bit depth in
>         the decoded buffer, game developers have told us in the past
>         that they
>         frequently use 22050Hz (or less) sounds in games, as well as
>         8/16 bits buffers.
>         Knowing their assets, only them can make the good call between
>         cutting on the
>         sample rate, the bit depth, or both.
>
>
>
>     This is a slightly different issue, namely "What's the lowest
>     quality I can accept for this asset?". I can see a value in giving
>     the Web dev the possibility to indicate that a given asset can use
>     a low quality internal representation (the exact syntax for this
>     has to be worked out, of course). The situation is somewhat
>     similar to how 3D APIs allow developers to use compressed textures
>     when a slight quality degradation can be accepted. For audio, I
>     think that sounds such as noisy or muddy sound effects could
>     definitely use a lower quality internal representation in many
>     situations. The same could go for emulators that mainly use
>     8/16-bit low-sample-rate sounds.
>
>     Having such an option in the API gives the implementation an
>     opportunity to save memory when memory is scarce, but it's not
>     necessarily forced to do so.
>
>     /Marcus
>
>
>     -- 
>     Marcus Geelnard
>     Technical Lead, Mobile Infrastructure
>     Opera Software
>
>


-- 
Marcus Geelnard
Technical Lead, Mobile Infrastructure
Opera Software