W3C home > Mailing lists > Public > whatwg@whatwg.org > March 2016

Re: [whatwg] VIDEO and pitchAdjustment

From: Garrett Smith <dhtmlkitchen@gmail.com>
Date: Tue, 8 Mar 2016 16:42:19 -0800
Message-ID: <CABZUbM0W+KjLY-SRWcTTCKmecHxT7p6pp5ZHNDhPpsORVcCJHg@mail.gmail.com>
To: Jer Noble <jer.noble@apple.com>
Cc: Philip Jägenstedt <philipj@opera.com>, Eric Carlson <eric.carlson@apple.com>, WHAT Working Group <whatwg@lists.whatwg.org>, Michael Enright <mike@kmcardiff.com>, Domenic Denicola <d@domenic.me>, "robert@ocallahan.org" <robert@ocallahan.org>
On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble <jer.noble@apple.com> wrote:
>
>> On Mar 4, 2016, at 3:19 PM, Garrett Smith <dhtmlkitchen@gmail.com> wrote:
>>
>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble <jer.noble@apple.com> wrote:
>>>
>>>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>
>>>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith <dhtmlkitchen@gmail.com> wrote:
>>>>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith <dhtmlkitchen@gmail.com>
>>>>>> wrote:
>>>>>>> On 11/12/15, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>>>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith <dhtmlkitchen@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 10/19/15, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>>>>>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt <philipj@opera.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola <d@domenic.me>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> From: Eric Carlson [mailto:eric.carlson@apple.com]
>>>>>
>>>>>> Two things.
>>>>>>
>>>>>> 1. Do the underlying media frameworks that browsers are using support
>>>>>> arbitrary pitch changes, or do they also only have the limited
>>>>>> preservesPitch-style API?
>>>>>
>>>>> Are there any problems getting in the way of pitch adjustment (without
>>>>> depending on playbackRate)?
>>>>
>>>> I don't know, that was basically my question too. If the underlying
>>>> APIs don't support it, that's a problem that needs to be fixed first.
>>>
>>>
>>> There are no such APIs on OS X which would allow an arbitrary pitch shift to be added to an otherwise normally playing piece of audio.
>>>
>>> IMO, this is a more appropriate request for the Web Audio API (adding a Audio Node which can add an arbitrary amount of pitch shift).  At which point, there would be no need for this in HTMLMediaElement, as authors could make a simple node graph consisting of an MediaElementAudioSourceNode and a PitchShiftNode.
>>>
>>> -Jer
>>
>> But that can't work on OSX, right?  I wonder how audio software on mac
>> does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
>> many others can all do this.
>
> None of them use the built in platform APIs to shift the pitch of encoded media.  Each does it manually, within their app, and each probably uses a different algorithm to achieve shift in pitch.
>
>> Plus how would web Audio API solve for the use case?
>>
>> To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
>> song is about a 50 cents flat of standard tuning. The pitch can be
>> adjusted by setting playbackRate to 1.023 and setting
>> MozPreservesPitch to false:—
>>
>> var vv = document.querySelector("video");
>> vv.mozPreservesPitch = 0;
>> vv.playbackRate = 1.023
>>
>> — but that speeds it up. I don't want speed coupled with pitch.
>
> The Web Audio equivalent would be:
>
> var video = document.querySelector(‘video’);
> video.preservesPitch = false;
> var context = new AudioContext();
> var sourceNode = context.createMediaElementSource(video);
> var pitchShiftNode = context.createPitchShift();
> pitchShiftNode.shiftAmount = 1.023;
> sourceNode.connect(pitchShiftNode);
> pitchShiftNode.connect(context.destination);
>

Which implementations does that work in?

That code is more complex than should be necessary. I see where you're
coming from on separating the audio. Could we move the media decorator
behind the scenes, and replace it with a simple getter/setter property
like `videoElement.audio` so that that can happen automagically?
Reminds me of createElement, createRange, document.implementation,
etc. Warts!

But then again, you also just said that there are no APIs on OS X that
allow an arbitrary pitch shift to be added to audio. If that is true,
then your `createPitchShift` code would be possible anyway, is that
right?

Thank you,
-- 
Garrett
@xkit
ChordCycles.wordpress.com
garretts.github.io
personx.tumblr.com
Received on Wednesday, 9 March 2016 00:42:47 UTC

This archive was generated by hypermail 2.4.0 : Wednesday, 22 January 2020 17:00:37 UTC