Re: Integer PCM sample formats to Web Audio API?

Hi again...

Actually, when looking into it, the SDL audio layer is quite low level. 
It's completely callback-driven (no support for channels, mixing, 
interpolation, etc), which means that you're not really utilizing the 
full potential of the Web Audio API.

SDL_mixer, on the other hand, has a higher level view of things that 
should be easier to map to the Web Audio API. If porting a big app from 
C to the Web I would personally build a more abstract version of the 
SDL_mixer API (e.g. by letting a Mix_Chunk represent an AudioBuffer 
rather than having an Uint8* buffer), and more or less forbid any use of 
the SDL_*Audio* methods and SDL_mixer callback based interfaces.

...just my 2 cents.

/Marcus


2013-12-02 14:40, Jukka Jylänki skrev:
> Hi,
>
> we have been working quite a lot with Web Audio API now while porting 
> native games to run on the web in the Emscripten community, and while 
> the experience has been quite good regarding the spec, there's 
> something that we feel that's missing that should definitely make its 
> way to the spec: adding support for integer sample types in addition 
> to Float32.
>
> The audio interfaces of the native world typically support a range of 
> different integer formats. For example, see the list of integer 
> formats in XAudio2 here: 
> http://msdn.microsoft.com/en-us/library/windows/desktop/microsoft.directx_sdk.ixaudio2.ixaudio2.createsourcevoice(v=vs.85).aspx 
> <http://msdn.microsoft.com/en-us/library/windows/desktop/microsoft.directx_sdk.ixaudio2.ixaudio2.createsourcevoice%28v=vs.85%29.aspx> 
> . The lack of these formats in Web Audio API has raised the following 
> issues:
>
>    - The common sample format for audio data in games tends to be 
> signed 16-bit. When porting large native games on the order of 
> magnitude like Epic Citadel ( http://www.unrealengine.com/html5/ ) we 
> have run into an issue, where having to use Float32 instead of 16-bit 
> ints doubles up the amount of memory that audio data needs. Since 
> these kind of games tend to use a lot of audio clips for different 
> sound effects, we have seen the Float32 requirement to consume 100MB+ 
> of extra memory for the browser, which is quite wasteful.
>
>    - Loading data from 16-bit int format requires either a load-time 
> conversion (Emscripten OpenAL support library does this), or a 
> run-time conversion when mixing audio (Emscripten SDL support library 
> does this). This requires a int-to-float conversion engine in JS like 
> this 
> https://github.com/kripken/emscripten/blob/df11c6f1fd1636a355b83a1c48b3a890596e6a32/src/library_sdl.js#L1725 
> , which is not an ideal operation to need to do in JS scripts.
>
>    - I have heard from Mozilla employees working on implemention Web 
> Audio API for Firefox for Android and FirefoxOS that not all mobile 
> devices support floating point audio playback in hardware, and that 
> browsers on Android might need to do a Float32->int conversion when 
> playing out data. In these cases we have integers on both sides of the 
> fence, but must talk in Float32 just for the spec.
>
> Having integer sample type support in the spec would let the pipeline 
> only have to do at most a single conversion in the browser C/C++ side, 
> and help memory pressure in large games and having to do any 
> potentially costly processing in JS. If this was convincing enough to 
> agree that Web Audio API should get integer sample formats, the next 
> question then is of course, which formats exactly should be added? I 
> would propose that at minimum the formats u8, s16 and s32 would be 
> included, possibly offering both signed and unsigned variants on each. 
> What do you think?
>
> Best Regards,
>    Jukka


-- 
Marcus Geelnard
Technical Lead, Mobile Infrastructure
Opera Software

Received on Monday, 2 December 2013 16:12:59 UTC