Re: Looking for a method to provide a 5+ second waveform preview as a user plays a <video>

Hi Eric,

First of all, sorry, but these kind of questions don't belong in this
mailing list, but rather on Stack Overflow or similar. This mailing list is
for feedback and discussion around the specs this working group provides.

Anyway, I'll give it a shot since you're on a good cause. ;)

On Wed, Sep 24, 2014 at 2:07 AM, Eric Wescott <wescotte@gmail.com> wrote:

>
> Why not have the ScriptProcessor gather up about 20 of these 16k buffers
> and then plot them all at once? Or just plot each 16k at the appropriate
> point in your graph?
>
> I don't quite understand how that would work... My understanding is
> "audioprocess" events dispatch at regular intervals by a ScriptProcessor
> and when I attempt to manually dispatch it via
>
> var event = new Event('audioprocess');
> for (var i=0; i < 20; i++)
>     audioScript.dispatchEvent(event);
>
> I get "Uncaught TypeError: Cannot read property 'getChannelData' of
> undefined" 20 or out 21 times from my function that handles "audioprocess"
> events.
>

Yes, because you're dispatching synthetic events and don't assign them the
inputBuffer and outputBuffer, like a native onaudioprocess event would. You
need to wait for the data, you can't rush the audio context to do the
processing when it's convenient for you. It pulls in data from the sources
when it's convenient for the audio context.


> However, even if that did work doesn't it just give you 16k samples based
> on the current audio being played? So if I dispatched 20 events in quick
> sucession would I just get 20 16 blocks with mostly overlapping sample data?
>
> Could you elaborate?
>

 I don't think I quite understood. Anyway, dispatching the events manually
doesn't really get you anything.

>
>
>>
>> I've also experimented with creating the entire waveform up front
>> <http://jsfiddle.net/daofw1g3/8/> which works but unfortunately has
>> issues with large files or longer audio clips.  I found if the video file
>> is > 1GB the FIle API dispatches an "error" event when I
>> readAsArrayBuffer(). So I thought perhaps I have to have the user isolate
>> the audio from the but that has it's own issues as well.  I tested it with
>> a 80minute 35mb mp3 and while ArrayBuffer is created but then
>> decodeAudioData() never runs it's callback function. I assume both cases
>> involve a memory limit?
>>
>
>  Well, 80 min of audio at 44.1 is like 800 MB of data. decodeAudioData
> should work, but it may take some time to decode it into memory. Assuming
> you have enough memory and process space.
>
>    I dunno I'm pretty convinced it's hitting some sort of memory limit.
> After file API creates the arrayBuffer my activity monitor shows no CPU,
> memory, or disk usage that suggests anything is actually going on and I let
> it sit for over a half hour and the callback is never executed...
>
>   If you don't want to do that, perhaps using a MediaElementAudioSource
> will work for you. It will stream the data into the audio graph where you
> can process it without having it all in memory.
>
>   Could you elaborate on this? Because I think that is what I'm already
> doing by way the below line...
>
> audioSource =
> audioContext.createMediaElementSource(document.getElementById("video"));
>
>
> Thank you for your patients as I'm still pretty new to javascript and the
> Web Audio API
>

I made an example of how to implement something like you want. Admittedly,
it's pretty naive as in it

1) Only works if the browser supports mp3 (because the file I had at most
handy was mp3) and ES6 Promises.
2) It's mono (probably not an issue for your use case).
3) Uses nearest-neighbour interpolation.
4) Doesn't resample on the fly (can be a serious memory problem when
analyzing long chunks of audio).

but it's just something I quickly threw together and hopefully it will more
give you an idea how to do this instead of being used as is. :)

Here's the gist:
https://gist.github.com/jussi-kalliokoski/edf81c322d0b49a97e98 and also a
demo page: http://fs.juss.in/waveform/

Keep in mind though that this will be a temporary solution, because the
ScriptProcessor will be deprecated in the future favor of a worker-based
solution.

- Jussi