Re: [mediacapture-main] Mandate implementations to support video frame subsampling (#723)

> (since different native resolutions may lead to different native frame rates).

Different pixel dimensions do require different frame rates. Greater pixel dimensions requires less frames per second than lesser pixel dimensions. That fact becomes immediately obvious when creating a video containing variable with and height frames, frame by frame. In that case a constant frame rate is not the goal, to avoid greater pixel dimension frames (less total frames required per second) to appear on screen rendered as fast as lesser pixel dimension frames (more total frames required per second), for example, see `console` at https://plnkr.co/edit/4Tb91b?preview note the different between the number of frames comparing index 0 and index 1, each set of frames being similar duration

`[Array(40), Array(194), Array(67), Array(102), Array(34), Array(32), Array(40)]`

using

```
const frames = await (await fetch(url)).json();
    console.log(frames);

    const rs = new ReadableStream({
      async pull(controller) {
        for (const frame of frames) {
          const [{
            duration, frameRate, width, height
          }] = frame;
          const framesLength = frame.length - 1;
          const frameDuration = Math.ceil((duration * 1000) / framesLength);
          for (let i = 1; i < framesLength; i++) {
            const response = await (await fetch(frame[i])).arrayBuffer();
            controller.enqueue({
              response, frameDuration
            });
          }
        }
        controller.close();
      }
    });

    const reader = rs.getReader();
    const processStream = async({
      value, done
    }) => {
      if (done) {
        await reader.closed;
        return "stream done";
      }
      const {
        response, frameDuration
      } = value;
      channel.postMessage({
        response
      }, [response]);
      await new Promise(resolve => setTimeout(resolve, frameDuration));
      return processStream(await reader.read());
    }
    const done = await processStream(await reader.read());
    postMessage(done);
  } catch (e) {
    console.error(e);
  }
```

which can also be done dynamically, frame by frame 

```
const
  flushFrame = (video) => {
    let
      now = video.currentTime;

    if (frameStartTime) {
      WebmWriter.addVideoFrame(
        canvas.toDataURL("image/webp").split(",").pop()
      , (now - frameStartTime) * 1000
      );
    }

    frameStartTime = now;
  };
  // ...
  frameStartTime = 0;
  const captureFrame = () => {
    captureTimer = null;
    flushFrame(video);
    ctx.drawImage(video, 0, 0);
    captureTimer = setTimeout(captureFrame, 1000 / inputSampleRate);
  };
  captureFrame();
```

https://plnkr.co/edit/ThXd9MKYvEYq2kKyh8oc?p=preview&preview

If variable width and height frames are not expected that setting a constant value for `width` and `height` is a possible viable feature, else would suggest testing the outcome of, for example setting `width` and `height` to `100`, respectively, where the input frame is `680` by `480`. What is the expected result?

There needs to be a clear distinction between _resizing_ and selecting _only part_ of the original frame - which does not appear to be be possible using any of the specified constaints.


-- 
GitHub Notification of comment by guest271314
Please view or discuss this issue at https://github.com/w3c/mediacapture-main/issues/723#issuecomment-691497185 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Saturday, 12 September 2020 14:23:52 UTC