Re: DSP API proposal/draft

Hey Marcus!

I'll take a deeper look at it later, but for now I have a few
questions/problems:
 * The API seems to be designed for most parts to deal with 1D/real data.
Imaginary/vector math is very core to DSP.
 * Most methods have no way of dealing with interleaved data. This is a
problem because a lot of data out there is in interleaved format, including
PCM. A few suggestions to alleviate the problem:
   - Add input & output stride & offset to the methods.
   - Add methods for deinterleaving/interleaving data (might be useful
anyway).
 * Since convolution is attached to the filter interface, it's also limited
to 1D signals, hence not very useful for images, etc.

Good work anyway!

Cheers,
Jussi

On Mon, Jul 23, 2012 at 4:08 PM, Marcus Geelnard <mage@opera.com> wrote:

> Hello again group!
>
> Hope you've had a nice weekend.
>
> I've put together a first (not yet complete) draft specification for a DSP
> API. I would like you (the W3 WG) to have a look at it and come with both
> high level feedback (is this a good idea to begin with?) and low level
> feedback (see points below).
>
> Here is the spec draft (as of 2012-Jul-23):
>
>   http://people.opera.com/mage/**dspapi/<http://people.opera.com/mage/dspapi/>
>
> Quite a bit of thought has gone into it, and some benchmarking has been
> done to find out what methods are useful to include, but I'm very open for
> discussions and changes. Here is a quick summary:
>
> * General
>  o There are three interfaces: DSP, FFT, Filter
>  o All interfaces are supposed to live in both the main context and in
> worker contexts.
>  o Only 1D signal processing has been considered/included.
>  o Only Float32Array is supported.
>  o Points for further discussion:
>    - Interface naming (e.g. name space bloat and collisions)?
>    - FFT & Filter are only useful as objects - should they be
> interface-less / only live on the DSP interface in some way (to minimize
> global name space pollution)?
>    - I believe that 2D and 3D signal processing is currently out of scope
> and risks bloating the API. Comments?
>    - We could possibly support Float64Array too, but that'd roughly double
> the complexity (code size, testing, etc), and I don't really see any real
> use cases for it in audio. Comments?
>
>
> * The FFT interface
>  o This is probably the most obvious one, since it has been discussed
> before and it's a key building block for lots of signal processing.
>  o Points for further discussion:
>    - The exact design of the interface - comments?
>
>
> * The Filter interface
>  o This is intended to implement both IIR and FIR filters of any order
> (inspired by the Scilab/Matlab filter function [1]).
>  o It supports cross-block filtering state.
>  o Points for further discussion:
>    - Should we simplify the case of supporting filtering of several
> channels with a single Filter object? E.g. have separate History objects
> (one for each channel) that you can manage yourself - or is the
> current interface good enough (e.g. manage your own copies of history
> arrays, or have one filter per channel)?
>
>
> * The DSP interface - general
>  o Most of the methods here are quite simple, and can easily be
> implemented in JavaScript directly.
>  o Benchmarks show that for most methods, we currently get a significant
> performance gain by using native methods even for simple operations, such
> as add(), for instance.
>  o I anticipate that many of the methods in the DSP interface will be
> possible to implement with native/close-to-native performance directly in
> JS at some point in the future, but it's quite risky to make assumptions
> about that (for Audio, we need stellar performance now).
>  o Points for further discussion:
>    - Is the interface bloated?
>    - Is it too restrictive?
>
>
> * The DSP interface - Math clones
>  o The Math object clones (sin, cos, sqrt, and friends) were all put on
> the DSP object for symmetry, even though some are obviously more
> useful/critical/important than others.
>
>
> * The DSP interface - Interpolation
>  o The sampleLinear() method uses non-uniform, edge-clamping sampling
> (inspired by the Scilab/Matlab interp1 function [2]).
>  o Non-uniform sampling can be used for implementing things such as:
>    - Traditional uniform sampling.
>    - Sweeping playback rate sampling.
>    - Looped playback (supply a looping time parameter).
>    - Wave-shaper (i.e. interpolate the shape curve using the input signal).
>  o Only linear interpolation has been defined.
>  o Points for further discussion:
>    - We probably want higher order interpolation too (cubic, sinc).
>    - Lower order interpolation too? (nearest)
>    - Should we support more specialized sampling methods (e.g. uniform
> sampling) too, to further improve performance?
>
>
> Regards,
>
>   Marcus
>
>
>
> [1] http://www.mathworks.se/help/**techdoc/ref/filter.html<http://www.mathworks.se/help/techdoc/ref/filter.html>
> [2] http://help.scilab.org/docs/5.**3.3/en_US/interp1.html<http://help.scilab.org/docs/5.3.3/en_US/interp1.html>
>
> --
> Marcus Geelnard
> Core Graphics Developer
> Opera Software ASA
>
>