Re: WebVR and DRM

On Mon, Jul 10, 2017 at 10:02 PM, David Singer <singer@apple.com> wrote:
>
> I don’t think WebGL (that you later mention) attempts to define time-vased
> behavior; certainly there is nothing about timestamps in the spec. So,
> indeed, using a layer that introduces some delay but doesn’t respect
> timestamps or even pass them through, is likely to lead to tricky timing
> problems. Why you assume this is necessarily true of WebVR is less clear.


I would like to address the "less clear" part about synchronization on an
example usecase that has actually occurred (and I believe will occur
again). In 2015 Underarmor published a WebGL experiment that composited
video and WebGL elements (http://gisele.underarmour.com/ now dead
unfortunately). You can watch a video of it here https://vimeo.com/105729794.
In order to do the compositing in realtime the video consisted of 3 parts.
A visibile part showing the actual video frame, a masking layer used to
separate foreground/background elements, and a data track (the bottom 10
pixel rows of the video) encoding a 3D tracking signal. How the masking
layer was used is obvious. The data track was read in in a shader (vertex
shader texture lookup) and used to construct a camera transformation
matrix, which was then used to properly position 3D elements to be
composited with the video.

It is quite likely people will want to composit/annotate 360° video in some
fashion. This composition will likely want to be properly synchronized with
the video, and one excellent way to do this would be such a hidden data
track on the video stream. It is an excellent solution because it avoids an
API entirely, or even reading back video frames into client memory, and
achieves perfect synchronization with the video frame that will be shown
directly on the GPU.

Of course I'd be remiss to point out that this fabulous technique would not
work with WebVR-DRM. We might want to think about that a bit...