[whatwg] Codecs for <audio> and <video>

As an aside to Chris McCormick's comments, I wonder if it might also be useful/possible/appropriate (or not) to provide access to media data in the way that the ActionScript computeSpectrum function does: 

http://livedocs.adobe.com/flash/9.0/ActionScriptLangRefV3/flash/media/SoundMixer.html#computeSpectrum%28%29

Sample visualization using Canvas with computeSpectrum: http://www2.nihilogic.dk/labs/canvas_music_visualization/

Sam Dutton

----------------------------------------------------------------------

Message: 1
Date: Sun, 9 Aug 2009 11:16:01 +1000
From: Silvia Pfeiffer <silviapfeiffer1@gmail.com>
Subject: Re: [whatwg] Codecs for <audio> and <video>
To: Chris McCormick <chris at mccormick.cx>
Cc: whatwg at lists.whatwg.org
Message-ID:
<2c0e02830908081816v74711d64ya72c8cc11550baa4 at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Aug 9, 2009 at 3:15 AM, Chris McCormick<chris at mccormick.cx> wrote:
> On Wed, Jul 08, 2009 at 09:24:42AM -0700, Charles Pritchard wrote:
>> There are two use cases that I think are important: a codec
>> implementation (let's use Vorbis),
>> and an accessibility implementation, working with a <canvas> element.
>
> Here are a few more use-cases that many people would consider just as
> important:
>
> * Browser based music software and synthesis toys.
> * New types of 'algorithmic' music like that pioneered by Brian Eno.
> * Browser based games which want to use procedural audio instead of
> pre-rendered sound effects.
>
> I'd like to reiterate the previously expressed sentiment that only implementing
> pre-rendered audio playback is like having a browser that only supports static
> images loaded from the server instead of animations and <canvas> tags.
>
> What is really needed is a DSP vector processor which runs outside of ECMA
> script, but with a good API so that the ECMAscripts can talk to it directly.
> Examples of reference software, mostly open source, which do this type of thing
> follow:
>
> * Csound
> * Supercollider
> * Pure Data
> * Nyquist
> * Chuck
> * Steinberg VSTs
>
> I am going to use the terms "signal vector", "audio buffer", and "array"
> interchangeably below.
>
> Four major types of synthesis would be useful, but they are pretty much
> isomorphic, so any one of them could be implemented as a base-line:
>
> * Wavetable (implement vector write/read/lookup operators)
> * FM & AM (implement vector + and * operators)
> * Subtractive (implement unit delay from which you can build filters)
> * Frequency domain (implemnt FFT and back again)
>
> Of these, I feel that wavetable synthesis should be the first type of synthesis
> to be implemented, since most of the code for manipulating audio buffers is
> already going to be in the browsers and exposing those buffers shouldn't be
> hugely difficult. Basically what this would take is ensuring some things about
> the audio tag:
>
> * Supports playback of arbitrarily small buffers.
> * Seamlessly loops those small buffers.
> * Allows read/write access to those buffers from ECMAscript.
>
> Given the above, the other types of synthesis are possible, albeit slowly. For
> example, FM & AM synthesis are possible by adding adding/multiplying vectors of
> sine data together into a currently looping audio buffer. Subtractive synthesis
> is possible by adding delayed versions of the data in the buffer to itself.
> Frequency domain synthesis is possible by analysing the data in the buffer with
> FFT (and reverse FFT) and writing back new data. I see this API as working as
> previously posted, by Charles Prichard, but with the following extra
> possibility:
>
> <audio id='mybuffer'>
> buffer = document.getElementById("mybuffer");
> // here myfunc is a function which will change
> // the audio buffer each time the buffer loops
> buffer.loopCallback = myfunc;
> buffer.loop = True;
> buffer.play();
>
> Of course, the ECMA script is probably going to be too slow in the short term,
> so moving forward it would be great if there was a library/API which can do the
> following vector operations in the background at a speed faster than doing them
> directly, element by element inside ECMAscript (a bit like Python's Numeric
> module). All inputs and outputs are signal vectors/audio tag buffers:
>
> * + - add two signal vectors (2 input, 1 output)
> * * - multiply two signal vectors (2 input, 1 output)
> * z - delay a signal vector with customisable sample length (2 input, 1 output)
> * read - do a table lookup (1 input, 1 output)
> * write - do a table write (2 input, 1 output)
> * copy - memcpy a signal vector (1 input, 1 output)
> * fft do a fast fourier transform - (1 input, 2 output)
> * rfft do a reverse fast fourier transform - (2 inputs, 1 output)
>
> It would be so great if it was possible to unify the above into an API that
> looked and worked something like this:
>
> <audio id='mybuffer'>
>
> outbuffer = document.getElementById("mybuffer");
>
> b = new AudioBuffer(64)
> for (x=0; x<64; x++)
> ? ? ? ?b[x] = Math.sin(x / 64 * Math.PI)a;
>
> // inside the loopCallback do a vector multiplication of the data in our buffer
> // with a sine wave we created earlier.
> outbuffer.multiply(b);
>

Why don't you just implement an example in javascript to show off what
you're talking about and make a use case for having it implemented
inside the browsers?

Cheers,
Silvia,

http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
					
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.whatwg.org/pipermail/whatwg-whatwg.org/attachments/20090809/db3df316/attachment-0001.htm>

Received on Sunday, 9 August 2009 07:23:15 UTC