Re: Integer PCM sample formats to Web Audio API?

Hello Jukka,

I think I see your points. Especially the issues of memory consumption 
and lack of floating point support in the back end (hardware/CPU) is 
worth considering, IMO.

On the other hand, I don't think that adding more types to the API is a 
good solution. As you said, there are many integer formats, and it's 
hard to know which of them are fast/supported in hardware, etc.

I would much rather see the integer:ism hidden under the hood. Even if 
the API gives you & requires Float32 at all points, the data could be in 
whatever format inside the audio engine. It would add an implicit 
conversion from/to Float32 for certain API entires, which would 
obviously come at a cost (shouldn't be more than a memcpy though), but 
on the other hand it could give (sometimes significant) speed & memory 
performance improvements (you could event support compressed audio 
buffers, similar to compressed textures in OpenGL/DX).

For the code-line that you indicated (convert interleaved U8/S16 buffers 
to de-interleaved F32 buffers), an alternate suggestion could be to 
extend the decodeAudioData method in the Web Audio API to allow its 
source to be "raw" data (i.e. header-less PCM), e.g. by passing an 
optional format specification object to the method (specifying data 
type, interleaving, channels and sample rate).

I'm not entirely sure how this proposal would improve your situation, 
though, nor if it's something that would gain support in this group 
(it's just an idea).

/Marcus



2013-12-02 14:40, Jukka Jylänki skrev:
> Hi,
>
> we have been working quite a lot with Web Audio API now while porting 
> native games to run on the web in the Emscripten community, and while 
> the experience has been quite good regarding the spec, there's 
> something that we feel that's missing that should definitely make its 
> way to the spec: adding support for integer sample types in addition 
> to Float32.
>
> The audio interfaces of the native world typically support a range of 
> different integer formats. For example, see the list of integer 
> formats in XAudio2 here: 
> http://msdn.microsoft.com/en-us/library/windows/desktop/microsoft.directx_sdk.ixaudio2.ixaudio2.createsourcevoice(v=vs.85).aspx 
> <http://msdn.microsoft.com/en-us/library/windows/desktop/microsoft.directx_sdk.ixaudio2.ixaudio2.createsourcevoice%28v=vs.85%29.aspx> 
> . The lack of these formats in Web Audio API has raised the following 
> issues:
>
>    - The common sample format for audio data in games tends to be 
> signed 16-bit. When porting large native games on the order of 
> magnitude like Epic Citadel ( http://www.unrealengine.com/html5/ ) we 
> have run into an issue, where having to use Float32 instead of 16-bit 
> ints doubles up the amount of memory that audio data needs. Since 
> these kind of games tend to use a lot of audio clips for different 
> sound effects, we have seen the Float32 requirement to consume 100MB+ 
> of extra memory for the browser, which is quite wasteful.
>
>    - Loading data from 16-bit int format requires either a load-time 
> conversion (Emscripten OpenAL support library does this), or a 
> run-time conversion when mixing audio (Emscripten SDL support library 
> does this). This requires a int-to-float conversion engine in JS like 
> this 
> https://github.com/kripken/emscripten/blob/df11c6f1fd1636a355b83a1c48b3a890596e6a32/src/library_sdl.js#L1725 
> , which is not an ideal operation to need to do in JS scripts.
>
>    - I have heard from Mozilla employees working on implemention Web 
> Audio API for Firefox for Android and FirefoxOS that not all mobile 
> devices support floating point audio playback in hardware, and that 
> browsers on Android might need to do a Float32->int conversion when 
> playing out data. In these cases we have integers on both sides of the 
> fence, but must talk in Float32 just for the spec.
>
> Having integer sample type support in the spec would let the pipeline 
> only have to do at most a single conversion in the browser C/C++ side, 
> and help memory pressure in large games and having to do any 
> potentially costly processing in JS. If this was convincing enough to 
> agree that Web Audio API should get integer sample formats, the next 
> question then is of course, which formats exactly should be added? I 
> would propose that at minimum the formats u8, s16 and s32 would be 
> included, possibly offering both signed and unsigned variants on each. 
> What do you think?
>
> Best Regards,
>    Jukka


-- 
Marcus Geelnard
Technical Lead, Mobile Infrastructure
Opera Software

Received on Monday, 2 December 2013 15:47:21 UTC