Re: [whatwg] VIDEO and pitchAdjustment

On Fri, Mar 11, 2016 at 1:19 PM, Jer Noble <jer.noble@apple.com> wrote:
>
> On Mar 11, 2016, at 1:11 PM, Garrett Smith <dhtmlkitchen@gmail.com> wrote:
>
>
>
> On Tuesday, March 8, 2016, Jer Noble <jer.noble@apple.com> wrote:
>>
>>
>> > On Mar 8, 2016, at 4:42 PM, Garrett Smith <dhtmlkitchen@gmail.com>
>> > wrote:
>> >
>> > On Fri, Mar 4, 2016 at 3:43 PM, Jer Noble <jer.noble@apple.com> wrote:
>> >>
>> >>> On Mar 4, 2016, at 3:19 PM, Garrett Smith <dhtmlkitchen@gmail.com>
>> >>> wrote:
>> >>>
>> >>> On Fri, Mar 4, 2016 at 1:55 PM, Jer Noble <jer.noble@apple.com> wrote:
>> >>>>
>> >>>>> On Mar 1, 2016, at 8:00 PM, Philip Jägenstedt <philipj@opera.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>> On Wed, Mar 2, 2016 at 9:19 AM, Garrett Smith
>> >>>>> <dhtmlkitchen@gmail.com> wrote:
>> >>>>>> On Thu, Nov 12, 2015 at 11:32 AM, Philip Jägenstedt
>> >>>>>> <philipj@opera.com> wrote:
>> >>>>>>> On Thu, Nov 12, 2015 at 10:55 AM, Garrett Smith
>> >>>>>>> <dhtmlkitchen@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>> On 11/12/15, Philip Jägenstedt <philipj@opera.com> wrote:
>> >>>>>>>>> On Thu, Nov 12, 2015 at 9:07 AM, Garrett Smith
>> >>>>>>>>> <dhtmlkitchen@gmail.com>
>> >>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> On 10/19/15, Philip Jägenstedt <philipj@opera.com> wrote:
>> >>>>>>>>>>> On Tue, Sep 1, 2015 at 11:21 AM, Philip Jägenstedt
>> >>>>>>>>>>> <philipj@opera.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>> On Mon, Aug 31, 2015 at 9:48 PM, Domenic Denicola
>> >>>>>>>>>>>> <d@domenic.me>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>> From: Eric Carlson [mailto:eric.carlson@apple.com]
>>
>>
>>
>>
>> >>
>> >> The Web Audio equivalent would be:
>> >>
>> >> var video = document.querySelector(‘video’);
>> >> video.preservesPitch = false;
>> >> var context = new AudioContext();
>> >> var sourceNode = context.createMediaElementSource(video);
>> >> var pitchShiftNode = context.createPitchShift();
>> >> pitchShiftNode.shiftAmount = 1.023;
>> >> sourceNode.connect(pitchShiftNode);
>> >> pitchShiftNode.connect(context.destination);
>> >>
>> >
>> > Which implementations does that work in?
>>
>> None, because there is no such thing as a PitchShiftNode.
>>
>
> I see.
>
>>
>> > That code is more complex than should be necessary. I see where you're
>> > coming from on separating the audio. Could we move the media decorator
>> > behind the scenes, and replace it with a simple getter/setter property
>> > like `videoElement.audio` so that that can happen automagically?
>> > Reminds me of createElement, createRange, document.implementation,
>> > etc. Warts!
>>
>> I’m not entirely sure what you’re asking here. If it’s that you don’t like
>> the `context.createMediaElementSource()` or `context.createPitchShift()`
>> syntax and would rather a constructor syntax, Issue #250
>> <https://github.com/WebAudio/web-audio-api/issues/250> in the Web Audio spec
>> is the issue for you.
>>
>> > But then again, you also just said that there are no APIs on OS X that
>> > allow an arbitrary pitch shift to be added to audio. If that is true,
>> > then your `createPitchShift` code would be possible anyway, is that
>>
>> There is no such API for such post-processing built into the OS X media
>> frameworks.
>
>
> Oh. Poor hardware integration, and now this…. Being Apple CEO is not Tim
> Cook's greatest gift…
>
>
> I’m not sure how that’s relevant.
>
>> As an example, there is an API for preserving pitch across rate changes:
>> -[AVPlayerItem setAudioTimePitchAlgorithm:]
>> <https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVPlayerItem_Class/index.html#//apple_ref/occ/instp/AVPlayerItem/audioTimePitchAlgorithm>.
>> So implementing the HTMLMediaElement “preservesPitch" property is trivial.
>> There is no such API for applying a specific and arbitrary amount of pitch
>> shifting. It would have to be implemented entirely by hand, the same way we
>> implement processing of media element audio with the Web Audio API.  Which
>> gets back to: at this point, we should just implement a Pitch Shift node in
>> Web Audio.
>>
>
>  I see.
>
>>> var video = document.querySelector(‘video’);
>>> var context = new AudioContext();
>>> var sourceNode = context.createMediaElementSource(video);
>>> var pitchShiftNode = context.createPitchShift();
>>> pitchShiftNode.shiftAmount = 1.023;
>>> sourceNode.connect(pitchShiftNode);
>>> pitchShiftNode.connect(context.destination);
>>>
>
> Too many steps and implementation details. It's complicated. It's asking for
> more js libraries and wrappers for that, and other situations related to
> videos' audio.  Make it simpler. Hide the implementation details.
>
> Hide:—
>  * AudioContext creation and subsequent video connection.
>  * createPitchShift — use audio.pitchShift to get a pitchShiftNode
>  * pitchShiftNode.shiftAmount — make pitchShiftNode a getter/setter (like
> video.playbackRate)
>
>>> var video = document.querySelector(‘video’);
>>> var context = new AudioContext();
>>> var sourceNode = context.createMediaElementSource(video);
>>> var context = new AudioContext();
>
> // Hidden AudioContext construction
> var audio = video.audio;
>
>>> var pitchShiftNode = context.createPitchShift();
>>> pitchShiftNode.shiftAmount = 1.023;
>
> //  Hidden AudioContext createPitchShift factory.
> //  Setter on pitchShift
> audio.pitchShift = 1.023;
>
>>> sourceNode.connect(pitchShiftNode);
>>> pitchShiftNode.connect(context.destination);
>>>
>
> Combined:—
>
> var video = document.querySelector(‘video’);
> video.audio.pitchShift = 1.023;
>
> The way you have it now is overkill when client doesn't want to care about a
> pitch shift node or other implementation details. And in that case, wouldn't
> it make sense to add  preservesPitch on the audio and deprecate it from the
> video?
>
>
> No, because preservePitch requires knowledge from the video element, namely
> the speed of playback.  An arbitrary pitch shift node needs no such
> information, and is useful outside of the context of a HTMLMediaElement.
>
> I also found:—
> https://developer.apple.com/library/prerelease/ios/documentation/AVFoundation/Reference/AVAudioUnitTimePitch_Class/
> ("preliminary document for an API or technology in development.”)
>
>
> You’ll be interested to know that the design of the Web Audio API was
> heavily influenced by the design of the OS X Core Audio library, and that is
> class you’ve found is an Objective-C wrapper around a Core Audio node.  So
> yes, this class could well be used to implement a Web Audio PitchShift node.
>

Sounds like you had to work around Apple's Core Audio. Some issues there?

Here's another example: In Heartbreaker solo of the great Jimmy Page,
the guitar tuning is different from the rest of the song.
https://www.youtube.com/watch?v=nwmCOSYUSlI

The audio is the only meaningful part of that video.

Thank you,
-- 
Garrett
Guitar Videos https://www.youtube.com/channel/UCY_uK9hur86KEuiG8s7yvWw
@xkit

Received on Wednesday, 23 March 2016 16:23:25 UTC