Re: Integer PCM sample formats to Web Audio API?

Hello Jukka,

I can see your point, and what you're saying is probably a good 
representation of the mindset of many [native/C++] game developers, so 
it is a real problem.

However, I would like to propose a slightly different mindset that I 
hope that serious game [engine] developers will come to adopt (with 
time?)...


2014-01-07 22:26, Jukka Jylänki skrev:
> Thanks for the replies!
>
> The SDL code I linked was only to serve as an example how such a 
> JS-implemented API currently looks like. The low-level SDL audio 
> doesn't itself store much audio data in memory, since it mixes on the 
> fly. The large memory usage cases come in when the browser stores the 
> data as Float32 internally after the audio buffers have been created, 
> which happens currently mostly with the OpenAL support library.
>
> 2013/12/2 Marcus Geelnard <mage@opera.com <mailto:mage@opera.com>>
>
>     Hi again...
>
>     Actually, when looking into it, the SDL audio layer is quite low
>     level. It's completely callback-driven (no support for channels,
>     mixing, interpolation, etc), which means that you're not really
>     utilizing the full potential of the Web Audio API.
>
>     SDL_mixer, on the other hand, has a higher level view of things
>     that should be easier to map to the Web Audio API. If porting a
>     big app from C to the Web I would personally build a more abstract
>     version of the SDL_mixer API (e.g. by letting a Mix_Chunk
>     represent an AudioBuffer rather than having an Uint8* buffer), and
>     more or less forbid any use of the SDL_*Audio* methods and
>     SDL_mixer callback based interfaces.
>
>     ...just my 2 cents.
>
>
>     /Marcus
>
>
> It is easy to mentally follow that route and say "API X is too 
> low-level" or "API X is not a good match for the target platform", 
> ergo for best practices API X should be banned when porting. Or that 
> "hundreds of clips consume 200MB+ of data" -> "why do you use so much 
> data simultaneously in memory? You should be more conservative", but 
> in these cases we are surrendering the concept of _porting_ and 
> instead talking about rewriting - or even worse, having to redesign 
> the application.
>
> The reason I am talking about porting is that the need for feature 
> parity arises most painfully especially when porting existing 
> projects: if native platform can push X audio clips in memory, and the 
> web build of the same app can only take in X/2 before needing to add 
> smartness (adding streaming, dropping an API that's a bad fit, 
> caching, compression, or whatever more intelligent solution one can 
> imagine), no matter how you look at it, the porting story is impacted. 
> When you factor in the more intelligent logic, the native platform is 
> still doubly better: it can cache twice the amount, stream twice as 
> much or compress twice the content. And if you happen to be porting a 
> project that was already caching, streaming and compressing, then 
> you're really in trouble since you don't know what you could do any 
> smarter, and you'll have no choice than to halve the content size, or 
> ask the user to buy double the RAM.
>
> Emscripten aims to develop a serious porting path all the way up to 
> the most demanding triple-A games out there. The majority of games 
> these days are developed using some kind of games engine middleware, 
> where the deployment story is equal to choosing the target platform 
> from a dropdown list and clicking Build. In these cases, porting is 
> not something temporary, transitional or a migration path between old 
> and new, but a permanent and standard day-to-day operation that is 
> done all the time. With such an ease for simultaneously targeting 
> multiple platforms, if one platform is not up to par, it stands out 
> very quick, and in a very bad way.

Targeting multiple platforms has always been, and will always be a 
tricky matter. If there's one thing that history has proven, it's that 
if you design a product for Platform A, and later decide to support 
Platform B, you'll be in trouble if you did not design the product with 
Platform B in mind from the start (there are numerous bad ports to show 
that).

An example is the situation with OpenGL vs DirectX (vs specific console 
3D APIs). For instance, it would be a really bad design to rely 
completely on DirectX in your code, and later slap on a DirectX->OpenGL 
wrapper layer in between the game code and the system. The proper design 
is to make your own intermediate 3D API (or use a 3rd party solution), 
that has optimized DirectX, OpenGL, OpenGL ES, etc back ends (possibly 
with slightly different rendering results in order to best utilize the 
different APIs). Many (most?) successful game engines support multiple 
graphics back ends, since game engine developers have learned that it's 
worth the effort.

What I mean is that you should keep the Web platform in mind when 
designing your game engine, otherwise you'll just be doing a port, and 
it is quite likely that the port will perform noticeably worse on the 
Web compared to the originally intended platform(s).

In the case of middleware software, it should recognize the APIs 
available on the Web platform, such as the Web Audio API, and have 
optimized back ends for those APIs, rather than using intermediate APIs 
(such as SDL) that will only generate sub-optimal API usage. I strongly 
feel that adapting the Web Audio API to the low level behavior of e.g. 
SDL is the wrong way forward here.

It's not just a problem of format conversions and memory overhead - it's 
also a problem of not utilizing the capabilities of the API, such as 
doing mixing and interpolation in JS instead of using the (possibly) 
SIMD-optimized and multi-threaded implementations in the Web Audio API.


>
> <hype-rant>The point I want to make is that C/C++ porting is not just 
> a fringe tech that only a few weird exotic game devs will be using. 
> Instead, crosscompiling C/C++ games will become the major tech that 
> both indie and professional games devs will use for developing any 
> kind of business related to games on the web. For games, audio is 
> equally as important as 3D rendering, and I (somewhat grandeously, I 
> know) claim that Emscripten is the single most important use case that 
> Web Audio has, and its Priority no.1 should be to take care that the 
> native-vs-web feature parity exists. As long as the parity is not 
> there and the web does not scale as well as native, HTML5 will lag 
> behind and the weird NPAPI/Flash/Java approaches we all so hate will 
> keep staying afloat.</hype-rant>
>
> I really really think that if native supports it, the web should 
> support it as well,

Agreed, but you have to have parity on the right level. There's no point 
in enforcing a design paradigm onto one platform just because it's a 
good fit for another platform.


/Marcus


>    Jukka
>
> 2013/12/2 Robert O'Callahan <robert@ocallahan.org 
> <mailto:robert@ocallahan.org>>
>
>     I think it would make sense to add an API to construct an
>     AudioBuffer from signed 16-bit PCM data. Then, as long as you
>     don't call getChannelData() on the buffer, the Web Audio
>     implementation can probably optimize its way to victory at least
>     for the simple cases.
>
>     Rob
>     -- 
>     Jtehsauts  tshaei dS,o n" Wohfy Mdaon  yhoaus  eanuttehrotraiitny 
>     eovni le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei
>     csha iids  teoa stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients 
>     uapr,e tfaokreg iyvoeunr, 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph
>     eann dt hwea lmka'n?  gBoutt uIp  waanndt  wyeonut  thoo mken.o w *
>     *
>
>


-- 
Marcus Geelnard
Technical Lead, Mobile Infrastructure
Opera Software