W3C home > Mailing lists > Public > whatwg@whatwg.org > March 2016

Re: [whatwg] VIDEO and pitchAdjustment

From: Jer Noble <jer.noble@apple.com>
Date: Tue, 08 Mar 2016 16:57:34 -0800
Message-id: <7B1276FE-B4B2-4600-AB22-6FA4260483C7@apple.com>
To: Garrett Smith <dhtmlkitchen@gmail.com>
Cc: Philip Jägenstedt <philipj@opera.com>, Eric Carlson <eric.carlson@apple.com>, WHAT Working Group <whatwg@lists.whatwg.org>, Michael Enright <mike@kmcardiff.com>, Domenic Denicola <d@domenic.me>, "robert@ocallahan.org" <robert@ocallahan.org>

> On Mar 8, 2016, at 4:42 PM, Garrett Smith <dhtmlkitchen@gmail.com> wrote:
> 
> On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble <jer.noble@apple.com> wrote:
>> 
>>> On Mar 4, 2016, at 3:19 PM, Garrett Smith <dhtmlkitchen@gmail.com> wrote:
>>> 
>>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble <jer.noble@apple.com> wrote:
>>>> 
>>>>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>> 
>>>>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith <dhtmlkitchen@gmail.com> wrote:
>>>>>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>>>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith <dhtmlkitchen@gmail.com>
>>>>>>> wrote:
>>>>>>>> On 11/12/15, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>>>>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith <dhtmlkitchen@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> On 10/19/15, Philip Jägenstedt <philipj@opera.com> wrote:
>>>>>>>>>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt <philipj@opera.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola <d@domenic.me>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> From: Eric Carlson [mailto:eric.carlson@apple.com]
>>>>>> 
>>>>>>> Two things.
>>>>>>> 
>>>>>>> 1. Do the underlying media frameworks that browsers are using support
>>>>>>> arbitrary pitch changes, or do they also only have the limited
>>>>>>> preservesPitch-style API?
>>>>>> 
>>>>>> Are there any problems getting in the way of pitch adjustment (without
>>>>>> depending on playbackRate)?
>>>>> 
>>>>> I don't know, that was basically my question too. If the underlying
>>>>> APIs don't support it, that's a problem that needs to be fixed first.
>>>> 
>>>> 
>>>> There are no such APIs on OS X which would allow an arbitrary pitch shift to be added to an otherwise normally playing piece of audio.
>>>> 
>>>> IMO, this is a more appropriate request for the Web Audio API (adding a Audio Node which can add an arbitrary amount of pitch shift).  At which point, there would be no need for this in HTMLMediaElement, as authors could make a simple node graph consisting of an MediaElementAudioSourceNode and a PitchShiftNode.
>>>> 
>>>> -Jer
>>> 
>>> But that can't work on OSX, right?  I wonder how audio software on mac
>>> does it, Audacity, Amazing Slow Downer, and Garage Band, Logic, and
>>> many others can all do this.
>> 
>> None of them use the built in platform APIs to shift the pitch of encoded media.  Each does it manually, within their app, and each probably uses a different algorithm to achieve shift in pitch.
>> 
>>> Plus how would web Audio API solve for the use case?
>>> 
>>> To frame an example, go to YT and pull up "Take it Easy" (Eagles). The
>>> song is about a 50 cents flat of standard tuning. The pitch can be
>>> adjusted by setting playbackRate to 1.023 and setting
>>> MozPreservesPitch to false:—
>>> 
>>> var vv = document.querySelector("video");
>>> vv.mozPreservesPitch = 0;
>>> vv.playbackRate = 1.023
>>> 
>>> — but that speeds it up. I don't want speed coupled with pitch.
>> 
>> The Web Audio equivalent would be:
>> 
>> var video = document.querySelector(‘video’);
>> video.preservesPitch = false;
>> var context = new AudioContext();
>> var sourceNode = context.createMediaElementSource(video);
>> var pitchShiftNode = context.createPitchShift();
>> pitchShiftNode.shiftAmount = 1.023;
>> sourceNode.connect(pitchShiftNode);
>> pitchShiftNode.connect(context.destination);
>> 
> 
> Which implementations does that work in?

None, because there is no such thing as a PitchShiftNode.

> That code is more complex than should be necessary. I see where you're
> coming from on separating the audio. Could we move the media decorator
> behind the scenes, and replace it with a simple getter/setter property
> like `videoElement.audio` so that that can happen automagically?
> Reminds me of createElement, createRange, document.implementation,
> etc. Warts!

I’m not entirely sure what you’re asking here. If it’s that you don’t like the `context.createMediaElementSource()` or `context.createPitchShift()` syntax and would rather a constructor syntax, Issue #250 <https://github.com/WebAudio/web-audio-api/issues/250> in the Web Audio spec is the issue for you.

> But then again, you also just said that there are no APIs on OS X that
> allow an arbitrary pitch shift to be added to audio. If that is true,
> then your `createPitchShift` code would be possible anyway, is that
> right?

There is no such API for such post-processing built into the OS X media frameworks. As an example, there is an API for preserving pitch across rate changes: -[AVPlayerItem setAudioTimePitchAlgorithm:] <https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVPlayerItem_Class/index.html#//apple_ref/occ/instp/AVPlayerItem/audioTimePitchAlgorithm>.  So implementing the HTMLMediaElement “preservesPitch" property is trivial.  There is no such API for applying a specific and arbitrary amount of pitch shifting. It would have to be implemented entirely by hand, the same way we implement processing of media element audio with the Web Audio API.  Which gets back to: at this point, we should just implement a Pitch Shift node in Web Audio.

-Jer

> Thank you,
> -- 
> Garrett
> @xkit
> ChordCycles.wordpress.com
> garretts.github.io
> personx.tumblr.com
Received on Wednesday, 9 March 2016 00:58:03 UTC

This archive was generated by hypermail 2.4.0 : Wednesday, 22 January 2020 17:00:37 UTC