Re: On diversity [was: Chrome Extension of Semantic Web]

> It's not just the input, it is also the output. You want to be able to emit triples in a way that can be safely forwarded to I/O, without overwhelming the receiver: that's the case Ruben and I have been discussing for some months :). In practice, you want congestion control/back-pressure the moment you want to import a decent number of triples into an async data storage.

> I think Ruben is working on an implementation of that idea.

I was, and it’s implemented now in node-n3 0.1.5:

var transform = new n3.Transform(),
    turtleStream = fs.createReadStream('cartoons.ttl');
turtleStream.pipe(transform);
transform.pipe(new SlowWriter());

function SlowWriter() {
  var writer = new Writable({ objectMode: true });
  writer._write = function (triple, encoding, done) {
    console.log(triple);
    setTimeout(done, 1000);
  };


Memory usage will stay low, even though SlowWriter only consumes one triple per second.
Reading from the input stream will only resume when the writer at the end of the chain is ready to receive more.

Best,

Ruben

Received on Monday, 14 October 2013 22:14:53 UTC