Re: Integer PCM sample formats to Web Audio API?

On Wed, Jan 15, 2014 at 5:01 PM, K. Gadd <kg@luminance.org> wrote:

> The most obvious analogue here is the way textures work in OpenGL and
> Direct3D. You allocate a texture of a particular size in a particular
> format, and that's what you get. The GPU is certainly free to take
> liberties with the actual arrangement of the texture in video memory (and
> in fact, most do), but the format you ask for is (IIRC) always the format
> you get. This is important because having extra format conversions
> introduced at the driver's discretion could result in unprecedented
> performance consequences or even behavioral differences (due to too
> much/too little precision).
>

If only that was true :-).
https://bugzilla.mozilla.org/show_bug.cgi?id=890950#c0


> I don't see how audio could really differ dramatically in this area,
> unless I've overlooked something important. I've love to see examples of
> how audio is somehow special in this regard.
>

I think the main difference is that audio is generally much lower
bandwidth/memory usage so conversions are more palatable. I know this is
not universally true.

P.S. in graphics scenarios we've been relying heavily on compressed storage
> of texel data in memory for over a decade, because it turns out we never
> have enough memory to store all our data. Given that the size of these
> float32 audiobuffers is problematic in reality for existing game demos,
> perhaps it could be worthwhile to use efficient in-memory compression for
> audio? It is certainly the case that lots of real-world games do streaming
> decompression for some of their audio (i.e. music and voiced dialogue)
> instead of decoding it up front into enormous buffers. Note that I am not
> advocating for streaming *from storage*, I am advocating for streaming
> *from memory*. The XBox 360 actually has support for this in the
> southbridge, if memory serves.
>

You can create a Blob from a typed array and feed it to through an <audio>
element connected to a MediaElementSourceNode. The Blob API does need to be
extended with an option to take ownership of (neuter) the typed array data.

Personally i support having decodeAudioData take an optional format
argument. You'd set it to "float" when you know you're going to call
getChannelData() anyway, and you'd set it to "short" when you want to
minimize memory usage, but by default it's up to the UA. We need a
parameter because the UA can't know which situation it's in. I'm guessing
it makes sense for UAs to default to "short' in most situations but I could
be wrong about that.

Of course the rule still applies that the UA can do anything it wants
that's observably equivalent to what you requested, so I don't support
writing into the spec some sort of implementation requirement that that
format actually be used internally, sorry.

Rob
-- 
Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
waanndt  wyeonut  thoo mken.o w