W3C home > Mailing lists > Public > public-audio@w3.org > July to September 2012

Re: DSP API proposal/draft

From: Marcus Geelnard <mage@opera.com>
Date: Mon, 23 Jul 2012 15:06:05 +0000
Message-ID: <20120723150605.gbs92neq889w40og@staff.opera.com>
To: Jussi Kalliokoski <jussi.kalliokoski@gmail.com>
Cc: public-audio@w3.org
Citerar Jussi Kalliokoski <jussi.kalliokoski@gmail.com>:

> Hey Marcus!
>
> I'll take a deeper look at it later, but for now I have a few
> questions/problems:

Thanks for the feedback!


>  * The API seems to be designed for most parts to deal with 1D/real data.
> Imaginary/vector math is very core to DSP.

Yes, I have not focused very much on complex data this far. This  
mostly has to do with:

- Audio signals are usually real valued, and I wanted to cover those  
use cases first.
- There's no natural representation of complex values in  
ECMAScript/Typed Arrays (e.g. we have to choose between interleaved  
arrays or multiple arrays).
- I wanted to keep the interface minimal at this point.

I think that many methods would work quite well with complex data, as  
long as you separate the real and imaginary parts into two different  
arrays and do two passes (e.g. for filtering, addition, etc). You can  
go from complex to polar with abs(real, imag) & atan2(imag, real) if  
you wish, and back again using cos(angle)*mag & sin(angle)*mag. Also,  
the FFT method supports complex input/output.

Are there any specific methods that you feel are missing?

>  * Most methods have no way of dealing with interleaved data. This is a
> problem because a lot of data out there is in interleaved format, including
> PCM. A few suggestions to alleviate the problem:
>    - Add input & output stride & offset to the methods.

I've thought about adding stride/offset/etc versions of all methods,  
but I fear that it would make the API complexity much higher (also  
from an implementation point of view).

>    - Add methods for deinterleaving/interleaving data (might be useful
> anyway).

Yes, this is a good idea! Do you have any method signatures in mind?  
Perhaps it would be a good idea to let these "data swizzling"  
operators support other kinds of typed arrays too (e.g. to go from  
Uint8Array with interleaved RGBA data to four Float32Array's)?

>  * Since convolution is attached to the filter interface, it's also limited
> to 1D signals, hence not very useful for images, etc.

Well, filtering, convolution, FFT and interpolation are all 1D. If we  
could agree on some common way to interpret 2D matrices in terms of  
typed arrays, we could add 2D versions of these methods (similar to  
the Matlab methods [1], [2] and [3], for instance).


Regards,

   Marcus


[1] http://www.mathworks.se/help/techdoc/ref/filter2.html
[2] http://www.mathworks.se/help/techdoc/ref/fft2.html
[3] http://www.mathworks.se/help/techdoc/ref/interp2.html


--
Marcus Geelnard
Core Graphics Developer
Opera Software ASA
Received on Monday, 23 July 2012 15:06:44 GMT

This archive was generated by hypermail 2.2.0+W3C-0.50 : Monday, 23 July 2012 15:06:46 GMT