Re: Web Audio API sequencer capabilities

Our emails crossed, of course!

OK, I see that you feel Srikumar's solution is good and I think that's fine
for now.

...joe
On Oct 5, 2012 6:53 PM, "Chris Rogers" <crogers@google.com> wrote:

>
>
> On Fri, Oct 5, 2012 at 2:30 PM, Joseph Berkovitz <joe@noteflight.com>wrote:
>
>> Chris,
>>
>> I don't see the expressivity of AudioParam as the main problem at all.
>> It's very well designed and very serviceable as an envelope generator. The
>> problem, if there is one, has to do with its inability to control multiple
>> nodes at the same time in an obvious way.
>>
>> If I could paraphrase Jussi here and perhaps distill one key point: it
>> seems difficult (although not impossible) to control *multiple* nodes with
>> a *single* scheduled sequence of parameter changes embodied in one
>> AudioParam, because every AudioParam belongs uniquely to the one AudioNode
>> that it controls. One very common synthesis use case would be the
>> simultaneous control of filter frequency and gain by the same ADSR
>> envelope, so that a note's timbre gets brighter at the same time that its
>> amplitude gets louder.
>>
>> Srikumar's approach -- I assume it actually works! -- seems as though it
>> will handle this case, but it doesn't feel like an obvious solution that
>> API consumers would quickly find their way to. Is this the recommended
>> approach to this use case?
>>
>
> Yes, I think Srikumar has already pointed out the solution to Jussi's
> problem.  If the intention is to control multiple AudioParams (living on
> different AudioNodes), then it's easy to create an AudioGainNode and use
> that as an envelope generator which can be connected to all of them.  And
> yes, this does work just fine :)  So, in other words, this envelope node
> does already exist in the form of an AudioGainNode.
>
> I would like to point out that the much more normal and common use case
> would be to control and automate a single AudioParam directly, so that's
> why the methods are included as a part of AudioParam.  It would be
> unfortunate for the simple cases to be required to create an extra node
> which must be connected to every parameter just to create a linear ramp.
>
> Chris
>
>
>
>
>>
>>
>> On Oct 4, 2012, at 2:46 AM, Jussi Kalliokoski <
>> jussi.kalliokoski@gmail.com> wrote:
>>
>> I think my intent is misinterpreted here, I know that you can do even
>> complex automation envelopes with the AudioParam, and its scheduling
>> capabilities are quite sufficient. What I'm saying is that the automation
>> behavior belongs to a separate node, and the AudioParam should be
>> simplified to have no scheduling capabilities, and if you want to schedule
>> its values you pipe the output of an envelope node into it.
>>
>> I talked a lot about the benefits and use cases of this in my initial
>> post about the subject [1], but one thing is a DAW plugin architecture,
>> where the plugin exposes the AudioParams necessary to control it. Imagine
>> we have a DAW that lets you draw envelopes for parameters, and a simple
>> oscillator plugin, that gives out just a detune parameter and uses the
>> built-in Oscillator. As Oscillators are single-shot, you can't give out the
>> detune AudioParam because it would just apply for one Oscillator. Hence it
>> probably makes sense to instead expose GainNodes that pipe their output to
>> the the respective AudioParams, but with that the enveloping possibilities
>> are lost. However, if there was a separate EnvelopeNode that handled the
>> value scheduling, the host could just pipe that to the GainNode that was
>> provided. It would also leave the plugin free to have its own value
>> scheduling for the detune if it has some tricks like frequency glide, and
>> those separately controlled automations would get mixed, and didn't have to
>> know about each other.
>>
>> Cheers,
>> Jussi
>>
>> [1] http://lists.w3.org/Archives/Public/public-audio/2012JulSep/0614.html
>>
>> On Thu, Oct 4, 2012 at 5:23 AM, Srikumar Karaikudi Subramanian <
>> srikumarks@gmail.com> wrote:
>>
>>> A gain node's gain parameter effectively serves as an envelope node if
>>> you feed a unity signal to the gain node. This has gotten really expressive
>>> particularly after connect() began supporting AudioParams as targets. Do
>>> you have a use case in mind that cannot be covered by such a gain node that
>>> would be covered by an envelope node?
>>>
>>>  -Kumar
>>>
>>> On 4 Oct, 2012, at 1:58 AM, Jussi Kalliokoski <
>>> jussi.kalliokoski@gmail.com> wrote:
>>>
>>> Let me be more specific, do you think the envelope functionality being
>>> in the AudioParam is more powerful than if it were in a separate node? If
>>> you do, why? What is the advantage it offers?
>>>
>>> Cheers,
>>> Jussi
>>>
>>> On Wed, Oct 3, 2012 at 8:40 PM, Chris Rogers <crogers@google.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Oct 3, 2012 at 12:59 AM, Jussi Kalliokoski <
>>>> jussi.kalliokoski@gmail.com> wrote:
>>>>
>>>>> Chris (Rogers), could I get your opinion regarding the introducing an
>>>>> envelope node and simplifying the AudioParam?
>>>>>
>>>>
>>>> AudioParam has been designed with lots of care and thought for
>>>> implementing envelopes, so I believe it's in a very good spot right now.
>>>>  As an example of how people are using these envelope capabilities in
>>>> sequencer applications, here's a good example from Patrick Borgeat:
>>>> https://dl.dropbox.com/u/15744891/www1002/macro_seq_test1002.html
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>> Jussi
>>>>>
>>>>>
>>>>> On Sat, Aug 25, 2012 at 8:19 AM, Srikumar Karaikudi Subramanian <
>>>>> srikumarks@gmail.com> wrote:
>>>>>
>>>>>> > This would be a very basic setup, but with the current API design
>>>>>> there are some hard problems to solve here. The audio is relatively easy,
>>>>>> regardless of whether it's coming from an external source or not. It's just
>>>>>> a source node of some sort. The sequencing part is where stuff gets tricky.
>>>>>>
>>>>>> Yes it does appear tricky, but given that scheduling with native
>>>>>> nodes suffices mostly, it seems to me that the ability to schedule JS audio
>>>>>> nodes using noteOn/noteOff (renamed now as start/stop), together with
>>>>>> dynamic lifetime support solves the scheduling problems completely. Such
>>>>>> scheduling facility need only be present for JS nodes that have no inputs -
>>>>>> i.e. are source nodes.
>>>>>>
>>>>>> We (at anclab) were thinking about similar scheduling issues within
>>>>>> the context of building compose-able "sound models" using the Web Audio
>>>>>> API. A prototype framework for this purpose that we built (
>>>>>> http://github.com/srikumarks/steller) will generalize if JS nodes
>>>>>> can be scheduled similar to buffer source nodes and oscillators. A bare
>>>>>> bones example of using the framework is available here -
>>>>>> http://srikumarks.github.com/steller .
>>>>>>
>>>>>> "Steller" is intended for interactive high level sound/music models
>>>>>> (think foot steps, ambient music generators and the like) and so doesn't
>>>>>> have time structures that are editable or even a "play position" as a DAW
>>>>>> would require, but it may be possible to build them atop/beside Steller. At
>>>>>> the least, it suggests the sufficiency of the current scheduling API for
>>>>>> native nodes.
>>>>>>
>>>>>> Best,
>>>>>> -Kumar
>>>>>>
>>>>>> On 21 Aug, 2012, at 11:28 PM, Jussi Kalliokoski <
>>>>>> jussi.kalliokoski@gmail.com> wrote:
>>>>>>
>>>>>> > Hello group,
>>>>>> >
>>>>>> > I've been thinking about how to use the Web Audio API to write a
>>>>>> full-fledged DAW with sequencing capabilities (e.g. MIDI), and I thought
>>>>>> I'd share some thoughts and questions with you.
>>>>>> >
>>>>>> > Currently, it's pretty straight-forward to use the Web Audio API to
>>>>>> schedule events in real time, which means it would play quite well together
>>>>>> with other real time APIs, such as the Web MIDI API. For example, you can
>>>>>> just schedule an audiobuffer to play whenever a noteon event is received
>>>>>> from a MIDI source.
>>>>>> >
>>>>>> > However, here's something of a simple idea of how to build a DAW
>>>>>> with a plugin architecture using the Web Audio API:
>>>>>> >
>>>>>> >  * You have tracks, which may contain audio and sequencing data
>>>>>> (e.g. MIDI, OSC and/or user-defined envelopes). All of these inputs can be
>>>>>> either being recorded from an external source, or be static pieces.
>>>>>> >
>>>>>> >  * You have an effects list for each track, effects being available
>>>>>> to pick from plugins.
>>>>>> >
>>>>>> >  * You have plugins. The plugins are given references to two gain
>>>>>> nodes, one for input and one for output, as well as a reference to the
>>>>>> AudioContext. In response, they will give AudioParam references back to the
>>>>>> host, as well as some information of what the AudioParams stand for,
>>>>>> min/max values and so on. The plugin will set up a sub-graph between the
>>>>>> given gain nodes.
>>>>>> >
>>>>>> > This would be a very basic setup, but with the current API design
>>>>>> there are some hard problems to solve here. The audio is relatively easy,
>>>>>> regardless of whether it's coming from an external source or not. It's just
>>>>>> a source node of some sort. The sequencing part is where stuff gets tricky.
>>>>>> >
>>>>>> > In the plugin models I've used, the sequencing data is paired with
>>>>>> the audio data in processing events, i.e. you're told to fill some buffers,
>>>>>> given a few k-rate params, a few a-rate params and some sequencing events
>>>>>> as well as the input audio data. This makes it very simple to synchronize
>>>>>> the sequencing events with the audio. But with the Web Audio API, the only
>>>>>> place where you get a processing event like this is the JS node, and even
>>>>>> there you currently only get the input audio.
>>>>>> >
>>>>>> > What would be the proposed solution for handling this case? And
>>>>>> please, no setTimeout(). A system is as weak as its weakest link and
>>>>>> building a DAW/Sequencer that relies on setTimeout is going to be utterly
>>>>>> unreliable, which a DAW can't afford to be.
>>>>>> >
>>>>>> > Cheers,
>>>>>> > Jussi
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>        ... .  .    .       Joe
>>
>> *Joe Berkovitz*
>> President
>>
>> *Noteflight LLC*
>> Boston, Mass.
>> phone: +1 978 314 6271
>>        www.noteflight.com
>>
>>
>