Re: Overlap between StreamReader and FileReader

Still thinking about it, my proposal is OK to handle simply delta data 
produced by the APIs, for the other proposals I could not see the link 
between the data producer APIs and the streams, then looking at what 
node does, maybe a createStream method could be added so we can do 
something like:

var fs1=new FileReader();
var ws1=new WebSocket();
var ws2=new WebSocket()
var xhr=new XHR();
var encrypt=crypto.subtle.encrypt(); //note: this does not work with the 
latest WebCrypto changes but let's say it does
var hash=crypto.subtle.digest();//idem
var specific_flow_control=new Stream();
var update_indexedDB=new Stream();
var msgpack=new Stream();

var stream_fs1=fs1.createStream();
var stream_ws1=ws1.createStream();
...
var stream_hash=hash.createStream();

stream_ws1.pipe(stream_encrypt).pipe(stream_ws2)

stream_xhr.pipe(stream_msgpack).pipe(update_indexedDB)

stream_ws1.pipe(stream_specific_flow_control).pipe(stream_xhr)

and maybe:

stream_fs1.pipe([stream_encrypt,stream_hash]).pipe([stream_ws2,update_indexedDB])

Regards

Aymeric

Le 13/09/2013 17:03, Aymeric Vitte a écrit :
>
> Le 13/09/2013 15:11, Takeshi Yoshino a écrit :
>> On Fri, Sep 13, 2013 at 9:50 PM, Aymeric Vitte 
>> <vitteaymeric@gmail.com <mailto:vitteaymeric@gmail.com>> wrote:
>>
>>
>>     Le 13/09/2013 14:23, Takeshi Yoshino a écrit :
>>>     Do you mean that those data producer APIs should be changed to
>>>     provide read-by-delta-data, and manipulation of data by js code
>>>     should happen there instead of at the output of Stream?
>>
>>     Yes, exactly, except if you/someone see another way of getting
>>     the data inside the browser and turning the flow into a stream
>>     without using these APIs.
>>
>>
>> I agree that there're various states and things to handle for each of 
>> the producer APIs, and it might be judicious not to convey such API 
>> specific info/signal through Stream.
>>
>> I don't think it's bad to convert xhr.DONE to stream.close() manually 
>> as in your example 
>> http://lists.w3.org/Archives/Public/public-webapps/2013JulSep/0453.html.
>>
>> But, regarding flow control, as I said in the other mail just posted, 
>> if we start thinking of flow control more seriously, maybe the right 
>> approach is to have unified flow control method and the point to 
>> define such a fine-grained flow control is Stream, not each API.
>
> Maybe, I was not at the start of this thread too so I don't know 
> exactly what was the original idea (and hope I am not screwing it up 
> here). But I am not sure it's possible to define a universal flow control.
>
> Example: I am currently experiencing some flow control issues for 
> project [1], basically the sender reads a file AsArrayBuffer from 
> indexedDB where it's stored as a Blob. Since we can not get delta data 
> while reading the File for now, the sender waits for having the whole 
> ArrayBuffer, then slices it, processes the blocks and sends them via 
> WebSockets. If you implement a basic loop, of course you overload the 
> sender's UA and connection. So the system makes some calculation in 
> order to allow only half of the bandwidth to be used, aggregate the 
> blocks until it finds out that the size of the aggregation meets the 
> bandwidth requirement for the aggregated blocks to be sent every 50ms.
>
> Then it uses a poor setTimeout to flush the data which screw up all 
> the preceding calculations since setTimeout fires whenever it likes. 
> Maybe there are smarter ways to do this, I was thinking to use workers 
> so you can get a more precise clock via postMessages but I did not try.
>
> In addition to the bandwidth control there is a window for flow control.
>
> I take this example to understand if this could be better with a 
> built-in Stream flow control, if so, after you have defined the right 
> parameters (if possible) for the streams flow control, you could 
> process delta data while reading the file and restream them directly 
> via WebSockets, and this would be great but again not sure that a 
> universal solution can be found.
>
>> If we're not, yes, maybe your proposal (deltaResponse) should be enough.
>
> What is sure is that delta data should be made available instead of 
> incremental ones.
>
> [1] http://www.peersm.com
> -- 
> jCore
> Email :avitte@jcore.fr
> Peersm :http://www.peersm.com
> iAnonym :http://www.ianonym.com
> node-Tor :https://www.github.com/Ayms/node-Tor
> GitHub :https://www.github.com/Ayms
> Web :www.jcore.fr
> Extract Widget Mobile :www.extractwidget.com
> BlimpMe! :www.blimpme.com

-- 
jCore
Email :  avitte@jcore.fr
Peersm : http://www.peersm.com
iAnonym : http://www.ianonym.com
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms
Web :    www.jcore.fr
Extract Widget Mobile : www.extractwidget.com
BlimpMe! : www.blimpme.com