API proposals for raw data - 1 - Introduction

**

*This is part 1 of a multipart posting*, *laying out various API ideas
in response to the need for "raw data access". *

*

Problem:


Many creative usages of WebRTC audio and video data require direct
access to the data - either in raw form or in encoded form.


All of these things are possible today, but they are not simple.


The current interfaces to get raw samples are:

  *

    Use WebAudio to generate samples as arrays of floats in WebAudio’s
    clock (example
    <https://webrtc.github.io/samples/src/content/getusermedia/volume/>)

  *

    Use Canvas to paint from a <video> tag (example
    <https://webrtc.github.io/samples/src/content/getusermedia/canvas/>)

For injecting raw data we have:

  *

    Use WebAudio to generate a MediaStreamTrack from raw audio data

  *

    Use canvas.captureStream() to capture frames from a Canvas (example
    <https://webrtc.github.io/samples/src/content/capture/canvas-video/>)

For getting encoded data we have:

  *

    MediaRecorder (example
    <https://webrtc.github.io/samples/src/content/getusermedia/record/>)

For injecting encoded data we have:

  *

    MSE feeding a <video> tag, and then generating a MediaStreamTrack
    from the <video> tag



All of these mechanisms are somewhat convoluted, they impose
restrictions on the form of data that can be delivered (often imposing
transcoding costs), and require the application writer to be familiar
with multiple shapes of API surface.


A simpler method is desirable.

*

-- 
Surveillance is pervasive. Go Dark.