W3C home > Mailing lists > Public > public-ldp@w3.org > June 2014

Re: Fwd: Re: Can we use TCP backpressure instead of paging?

From: Andreas Kuckartz <a.kuckartz@ping.de>
Date: 11 Jun 2014 11:20:03 +0200
Message-ID: <53981F43.3000301@ping.de>
To: public-ldp@w3.org
These communities could and should be involved in addition to LDP:
- http
- xmpp
- W3C Social Interest Group (which hopefully starts in a few days)
- Hydra
- Linked Data Fragments

Some of them potentially can influence browser manufacturers.

I also think that this might be relevant for the IoT.


David Booth:
> Wow, very intriguing idea!  This really seems worth further
> investigation.  If this approach works out it could address the whoel
> paging issue across the board rather than piecemeal by each spec.
> David
> On 06/10/2014 02:03 PM, Sandro Hawke wrote:
>> On 06/10/2014 01:34 PM, Andreas Kuckartz wrote:
>>> -------- Original Message --------
>>> To: Sandro Hawke <sandro@w3.org>
>>> CC: Linked Data Platform WG <public-ldp-wg@w3.org>
>>> Thanks a lot for thinking outside the current box of paging. I have
>>> looked at several paging approaches and do not really like any of them.
>>> They contaminate the real data and/or seem to be unnecessarily complex
>>> to implement.
>> You're welcome.     I did a little more investigation today, including
>> writing a tiny node.js server that streams data and lets me see what
>> what happens when I do things on the client.    This little bit of
>> testing was encouraging.
>> I also spoke to a few people at lunch here at the W3C AC meeting and
>> again no one saw any serious problem.
>> No clue yet how likely it is that browser vendors might implement an
>> extension to xhr to allowed WebApps to use this properly.
>>          -- Sandro
>>> Cheers,
>>> Andreas
>>> ---
>>> Sandro Hawke:
>>>> Thinking about paging a little, I'm really wondering if one isn't
>>>> better
>>>> off using TCP backpressure instead of explicit paging.  It would have
>>>> the huge advantage of requiring little or no special code in the client
>>>> or the server, if they already implement high-performance streaming.
>>>> (I started thinking about this because as far as I can tell, if we want
>>>> to allow LDP servers to initiate paging, we have to require every LDP
>>>> client to understand paging.   That's a high bar.   If you want to
>>>> respond to that particular point, please change the subject line!)
>>>> The key point here is that TCP already provides an elegant way to
>>>> handle
>>>> arbitrarily large data flows to arbitrary small devices on poor
>>>> connections.    If someone knows of a good simple explanation of this,
>>>> please send along a pointer.   My knowledge is largely pre-web.
>>>> In web software we often to think of HTTP operations as atomic steps
>>>> that take an arbitrary long time.   With that model, doing a GET on a
>>>> 100G resource is pretty much always a mistake.  But nothing in the web
>>>> protocols requires thinking about it that way.   Instead, one can think
>>>> of HTTP operations as opening streams which start data flowing.
>>>> In some situations, those streams will complete in a small number of
>>>> milliseconds, and there was no advantage to thinking of it as a stream.
>>>>    But once you hit human response time, it starts to make sense to be
>>>> aware that there's a stream flowing.     If you're a client doing a
>>>> GET,
>>>> and it's taking more than maybe 0.5s, you can provide a better UX by
>>>> displaying something for the user based on what you've gotten so far.
>>>> What's more, if the app only needs the first N results, it can read the
>>>> stream until it gets N results, then .abort() the xhr.   The server may
>>>> produce a few more results than were consumed before it knows about the
>>>> .abort(), but I doubt that's too bad in most cases.
>>>> The case that's not handled well by current browsers is pausing the
>>>> stream.   In theory, as I understand it (and I'm no expert), this
>>>> can be
>>>> done by simply using TCP flow control.   A non-browser app that stops
>>>> reading data from its socket will exert backpressure, eventually
>>>> resulting in the process writing data finding the stream's not ready
>>>> for
>>>> writing.   My sense is that can and does work rather well in a wide
>>>> variety of situations.
>>>> Unfortunately, as I understand it, this doesn't work in WebApps today,
>>>> because the browser will just keep reading and buffering until it runs
>>>> out of VM.   If instead xhr (and websockets) had a limit on how much it
>>>> would buffer, and webapps could set that (and probably it starts around
>>>> 10M), then a WebApp that stopped consuming data would produce
>>>> backpressure that would result in the server learning it can't send any
>>>> more yet.     When the WebApp consumes more data, the server can start
>>>> sending again.
>>>> I'm very curious if there's any technical reason this wont work.   I
>>>> understand there may be problems with some existing software, including
>>>> browsers, not handling this kind of streaming.  But is there
>>>> anything in
>>>> the basic internet protocols and implementations that make this not
>>>> work?     For instance, it may be that after blocking for a long time
>>>> (minutes, waiting for the user to request more), restarting is too
>>>> slow,
>>>> or something like that.
>>>> One possible workaround for the lack of browser support would be for
>>>> servers to be a bit smarter and make some guesses.  For example, a
>>>> server might say that requests with User-Agent being any known browser
>>>> should be served normally for the first 10s, then drop to a much slower
>>>> speed, consuming resources in the server, the net, and the client.
>>>> WebApps that want to sidestep this could do so with a Prefer header,
>>>> like Prefer initial-full-speed-duration=1s or 1000s.    At some point,
>>>> when browsers allow webapp backpressure, those browser User-Agent
>>>> strings could be exempted from this slowdown.
>>>>       -- Sandro
Received on Wednesday, 11 June 2014 09:45:43 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 16:16:37 UTC