Re: ACTION-11: availability of protected media through JS APIs

On Mon, May 13, 2013 at 11:09 AM, Mark Watson <watsonm@netflix.com> wrote:

> All,
>
> ACTION-11 arose from the discussion on 3/26 [1] and bug 21155 [2].
>
> It's clear that there may be cases where media data is not available to
> the Javascript. Specifically when media decryption, decoding and rendering
> are all performed in a protected media pipeline. It's also possible that
> CDMs place restrictions on what a UA can do with media data even in the
> case that the decoded media data is passed back to the UA (what
> restrictions a UA implements on behalf of a CDM is something determine
> during UA/CDM integration and not something we need to standardize).
>
> In looking at the JS APIs, there are two possible ways this could be
> handled:
> (a) media data doesn't (logically) leave the media element at all, and so
> these JS APIs cannot be used at all
> (b) media data can be processed using Canvas and Audio APIs, but the raw
> bytes of the media cannot be returned to JS
>
> I expect that in the short term (generally) and medium term (for hardware
> protected pipelines), then (a) is most likely to be implemented. It seems
> that the specification should not preclude (b), though.
>
> We need to describe what happens when JS APIs for access to media are used
> in these cases. I propose adding a small section (position tbd) as shown
> below, which is based on these methods silently behaving as if there was no
> media data. Alternatively, we could introduce explicit error cases where
> the unavailability of the media data is detectable by the script.
>
> "X.Y Availability of media data through Javascript APIs
>
> Media data processed by a CDM may not be available through Javascript APIs
> in the usual way.
>

Suggest dropping "in the usual way" since that merely begs the question of
"what is the usual way?" which I think you don't want to specify.


> This specification does not define conditions for such non-availability of
> media data, however, if media data is not available to Javascript APIs then
> these APIs MAY behave as if no media data was present at all. For the
> purpose of the following we call an HTMLMediaElement making use of a CDM a
> "protected media element".
>


> Specifically:
>
> - the CanvasRenderingContext2D drawImage method [3], when invoked with a
> protected media element, MAY behave as if the media element readyState was
> HAVE_NOTHING. Note: according to [3] this implies that nothing will be
> drawn.
> - the AudioContext MediaElementAudioSourceNode [4], MAY behave as if
> a protected media element has no audio track. Note: this should also imply
> that the usual createMediaElementSource() behaviour of disabling the normal
> media element audio rendering would *not* apply.
>
> Alternatively, these APIs may perform their usual functions, except that
> the media data itself may not be available to Javascript APIs. Specifically:
> - when the CanvasRenderingContext2D drawImage method is used to render a
> video frame to a Canvas, then subsequent use of the getImageData() method
> on this Canvas or on any other Canvas to which data from the original
> Canvas has been drawn MAY return image data that is entirely transparent
> black.
> - an AudioContext ScriptProcessorNode MAY not fire the onaudioprocess
> event if any earlier node in the audio processing graph is a
> MediaElementAudioSourceNode created from a protected media element."
>

I wonder if we should endeavor to normatively define this behavior in EME
or ask the relevant specs to be modified accordingly. I would prefer the
latter. Doing it in EME creates a precedent and sets expectations that EME
will address new situations in the future, or risk being read as not
potentially incurring the same constraints.

However, if it is thought necessary to include in EME, then I think we will
need to also say something about the possibility of text tracks/cues being
unavailable, and possibly other metadata, like the number of video tracks
or the number of audio tracks present, their @kind, etc.


>
> ...Mark
>
> [1] http://www.w3.org/2013/03/26-html-media-minutes.html#item05
>
> [2] https://www.w3.org/Bugs/Public/show_bug.cgi?id=21155
>
> [3]
> http://www.w3.org/html/wg/drafts/2dcontext/html5_canvas/#dom-context-2d-drawimage
>
> [4] http://www.w3.org/TR/webaudio/#MediaElementAudioSourceNode-section
>