Re: [whatwg/streams] Stream Reader: event to detect first byte added to internal buffer (#1126)

@ricea thanks for the quick response! Yes, the reason of this issue is actually to point out the lack of an event signaling the start of the chunk transfer via network for each chunk separately. Without such information correct throughput measurement seems impossible.

Let me provide a minimal self-contained code example to showcase the problem:

Node.js script to produce chunked-transfer data and serve the index.html below
```javascript
const http = require('http');
const fs = require('fs');

let index = '';
fs.readFile('index.html', (err, data) => {
    if (err) {
        throw err;
    }
    index = data.toString();
});

const hostname = '127.0.0.1';
const port = 3000;

async function produceData() {
    return new Promise(resolve => {
        setTimeout(resolve.bind(this, Buffer.alloc(1024)), 100);
    });
}

const server = http.createServer(async (req, res) => {
  res.statusCode = 200;
  if (req.url !== '/data') {
    res.setHeader('Content-Type', 'text/html');
    res.end(index)
    return;
  }
  res.setHeader('Content-Type', 'application/octet-stream');
  res.setHeader('Transfer-Encoding', 'chunked');
  
  for (let iter = 0; iter < 100; iter++) {
    const chunk = await produceData();
    res.write(chunk);
  }
  res.end();
});

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});
```

HMTL (index.html)
```html
<!DOCTYPE html>
<html lang="en">
    <head>
        <title>fetch api - stream reader - throughput check</title>
    </head>
    <body>
        please open dev tools

        <script>
            fetch('./data')
                    .then(response => response.body)
                    .then(body => {
                        const reader = body.getReader();

                        let timeMark = Date.now();
                        let timeSum = 0;
                        let byteSum = 0;
                        let chunkCount = 0;
                        function pump() {
                            return reader.read().then(({ done, value }) => {
                                if (done) {
                                    console.log(`got all chunks. Prize question: what is the actual network throughput? ${byteSum / (timeSum / 1000)} bytes per second does not seem right!`)
                                    return;
                                }
                                console.log(`got ${++chunkCount}. chunk with ${value.byteLength} bytes, in ${Date.now() - timeMark} ms`);
                                byteSum += value.byteLength;
                                timeSum += (Date.now() - timeMark);
                                timeMark = Date.now();
                                return pump();
                            });
                        }
                        pump();
                    })
        </script>
    </body>
</html>
```

If you start above node script and navigate to `http://127.0.0.1:3000/` you should see the following in dev tools
```
got 1. chunk with 1024 bytes, in 0 ms
got 2. chunk with 1024 bytes, in 103 ms
got 3. chunk with 1024 bytes, in 101 ms
...
got 98. chunk with 1024 bytes, in 101 ms
got 99. chunk with 1024 bytes, in 103 ms
got 100. chunk with 1024 bytes, in 100 ms
got all chunks. Prize question: what is the actual network throughput? 10123.578843302026 bytes per second does not seem right!
```

So, did i misunderstood something or is it simply impossible to measure correct network throughput with ```fetch``` api and the stream reader?

Side note: For more clarity i've changed the issue title from
"Stream Reader: event to detect first byte added to internal buffer"
to
"Stream Reader: event to detect first byte of each chunk added to internal buffer"
and desired event name to 'chunkTransferStarted'. 








-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/streams/issues/1126#issuecomment-839151920

Received on Tuesday, 11 May 2021 20:46:40 UTC