- From: Jesse Thompson <jesse.thompson@wisc.edu>
- Date: Wed, 16 Apr 2014 09:22:50 -0500
- To: w3c-dist-auth@w3.org
- Cc: tc-push-l@lists.calconnect.org
I chimed in on this conversation on the tc-push-l@lists.calconnect.org list. Cyrus asked me to post my comments here. (I tried to refactor content from multiple messages for context, but I apologize if it is still a bit jumbled - and for top posting) I don't have much knowledge in implementing push/realtime HTTP other than installing various web clients on top of Ejabberd using XEP-0206 (XMPP over BOSH) and deciding not to use XEP-0025 (Jabber XMPP Polling at UW-Madison. http://xmpp.org/extensions/xep-0206.html http://xmpp.org/extensions/xep-0025.html Otherwise, what I am saying is based on what I have read or what others have told me, so take it with a grain of salt. I suggested looking at the work being done on websockets for XMPP. Not that XMPP should be used as the protocol, but rather that they might have documented some of the same hurdles you are trying to overcome. http://tools.ietf.org/html/draft-ietf-xmpp-websocket-02 From my standpoint as a calendar server administrator, I shudder to think of the impact of having tens of thousands of clients keeping connections open to my main Apache preforking servers. The root of my recommendation is not to say whether long polling is better or worse than web sockets, but rather to suggest the option to have the clients make these long running requests to a secondary URI. This would allow server administrators to handle those request with a web server specifically optimized for that type of connection. In response to Brad, who brought up the topic of what data is sent over the websocket, I said that I would consider it an implementation detail as to what information (and how much) to send over the web socket connection vs. telling the client to sync via the normal connection. The people I know who have implemented realtime web apps generally ditch the idea of long polling because it doesn't scale. They use web sockets, but they have clients connect on a different connection than the main web server that is serving normal requests. Not all web servers are non-blocking/asynchronous, so you might not be able/willing to completely replace existing web servers and downstream reverse proxy servers. The 2-web server strategy allows you to add in push capabilities while remaining backwards compatible, as well as segregate load. If your web sockets server becomes overloaded, clients should be able to fall back on the non-push capabilities. For an example, I know the guys that built https://www.thegamecrafter.com/ Their main web servers are not handling any of the web sockets connections. But their application is still capable of pushing real time updates. They accomplish this by having the client (javascript) maintain a connection to https://www.firebase.com/ to receive those updates. Firebase is an example of a cloud hosting service for realtime apps. It allows you to outsource the realtime/push capabilities of your apps. I don't know if something like that would be appropriate for calendaring, but it might be an instructive exercise to proof-of-concept leveraging this type of solution for your work on tc-push. It might at least lead you to the answers to the questions your were asking. Anyway, I hope this is helpful information. -- Jesse Thompson Technical Consultant - Messaging Productivity and Collaboration Solutions Division of Information Technology University of Wisconsin–Madison On 4/9/2014 11:28 AM, Ken Murchison wrote: > All, > > The Calendaring and Scheduling Consortium (CalConnect) is looking at > ways to have a server "push" changes made to a calendar/addressbook > collection out to a client. There are already a few proprietary > mechanisms in place for doing so, but we would like to come up with > something standard that would be relatively simple to implement for both > clients and servers and would be applicable to any DAV collection. > > One idea that we are toying with is to leverage the existing > DAV:sync-collection REPORT <http://datatracker.ietf.org/doc/rfc6578/> > and HTTP long-polling <http://datatracker.ietf.org/doc/rfc6202/>. I > spent a few hours coding up a prototype version of HTTP long-polling and > HTTP streaming for DAV:sync-collection REPORTs in my server which I > describe below. We (CalConnect) are considering using this approach or > something similar as a starting point and we are interested in any/all > feedback from the larger DAV community, including: > > * Is this approach sane, is there a better way, or is any type of push > via HTTP a hopeless endeavor? > * Will this approach (or anything similar) break in the face of > intermediaries? > * Will existing HTTP/DAV stacks be able to handle long-polling and/or > streaming? > * Should the server advertise its ability to long-poll and/or stream > for the client to discover or simply leave it up to the client try > one or both and see what the server does, as is the case in my > implementation below? > > > Long-polling: > > For long-polling, I leveraged the HTTP Prefer header > <http://tools.ietf.org/html/draft-snell-http-prefer> and its 'wait' > preference as a way for the client to tell the server that it wants to > long poll. If the client doesn't specify a DAV:sync-token (initial > sync) or if there have been changes since the specified token, then the > server will respond immediately. Otherwise, it will only respond if a > change is detected or when the timeout expires, whichever comes first. > In the case of a delayed response, I issue a 100 (Continue) provisional > response with a Preference-Applied header to notify the client that the > server is indeed long-polling as requested. This provisional response > may or may not be necessary. In my implementation I wait 1 sec less > than specified to account for processing time so I don't go over what > the client expects. > > Streaming: > > The client can request streaming behavior by simply including an Accept > header with the 'multipart/mixed' media type (I chose this subtype for > lack of something better - we could use the existing x-mixed-replace or > create our own). The client can also specify a timeout for the > streaming using the same 'wait' preference as used for long-polling. In > the absence of a client-requested timeout, the server will continue to > add body parts until the client disconnects or the server hits some > internal timeout. Because a multipart response allows for an epilogue > following the final delimiter, a client can't just rely on the > delimiters to detect the end of the response. Therefore, the server > MUST use either chunked TE or close the connection following the > multipart response. In my example below, I close-delimit the response > for better readability. FWIW, I think the same holds true for > multipart/byteranges responses. In my implementation I include a > Content-Length header in the body-part headers so the client can detect > the end of the XML body without looking for the trailing delimiter > (mainly because the trailing delimiter doesn't appear until the next > body part). > > Examples: > > Here is an example of long-polling with 3 requests (I added superfluous > Date headers to show the timing). The first request returns immediately > due to a pre-existing change, the second returns upon detecting a > subsequent change some 95 sec later, and the third times out after 3 min. > > REPORT /dav/calendars/user/ken/Default/ HTTP/1.1 > Host: localhost > Date: Wed, 30 Oct 2013 18:11:11 GMT > Content-Type: application/xml > Content-Length: 260 > Prefer: return=minimal, wait=180 > > <?xml version="1.0" encoding="UTF-8"?> > <C:sync-collection xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-354</D:sync-token> > <D:sync-level>1</D:sync-level> > <D:prop/> > </C:sync-collection> > > HTTP/1.1 207 Multi-Status > Date: Wed, 30 Oct 2013 18:11:11 GMT > Vary: Accept-Encoding, Brief, Prefer > Preference-Applied: return=minimal, wait=180 > Content-Type: application/xml; charset=utf-8 > Content-Length: 421 > > <?xml version="1.0" encoding="utf-8"?> > <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:response> > <D:href>/dav/calendars/user/ken/Default/4E4B3490-6F01-41B9-AA5B-FE2CD6A30632.ics</D:href> > <D:propstat> > <D:prop/> > <D:status>HTTP/1.1 200 OK</D:status> > </D:propstat> > </D:response> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-355</D:sync-token> > </D:multistatus> > > > > REPORT /dav/calendars/user/ken/Default/ HTTP/1.1 > Host: localhost > Date: Wed, 30 Oct 2013 18:11:12 GMT > Content-Type: application/xml > Content-Length: 260 > Prefer: return=minimal, wait=180 > > <?xml version="1.0" encoding="UTF-8"?> > <C:sync-collection xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-355</D:sync-token> > <D:sync-level>1</D:sync-level> > <D:prop/> > </C:sync-collection> > > HTTP/1.1 100 Continue > Date: Wed, 30 Oct 2013 18:11:12 GMT > Preference-Applied: wait=180 > > HTTP/1.1 207 Multi-Status > Date: Wed, 30 Oct 2013 18:12:47 GMT > Vary: Accept-Encoding, Brief, Prefer > Preference-Applied: return=minimal, wait=180 > Content-Type: application/xml; charset=utf-8 > Content-Length: 375 > > <?xml version="1.0" encoding="utf-8"?> > <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:response> > <D:href>/dav/calendars/user/ken/Default/4E4B3490-6F01-41B9-AA5B-FE2CD6A30632.ics</D:href> > <D:status>HTTP/1.1 404 Not Found</D:status> > </D:response> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-356</D:sync-token> > </D:multistatus> > > > > REPORT /dav/calendars/user/ken/Default/ HTTP/1.1 > Host: localhost > Date: Wed, 30 Oct 2013 18:12:48 GMT > Content-Type: application/xml > Content-Length: 260 > Prefer: return=minimal, wait=180 > > <?xml version="1.0" encoding="UTF-8"?> > <C:sync-collection xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-356</D:sync-token> > <D:sync-level>1</D:sync-level> > <D:prop/> > </C:sync-collection> > > HTTP/1.1 100 Continue > Date: Wed, 30 Oct 2013 18:12:48 GMT > Preference-Applied: wait=180 > > HTTP/1.1 207 Multi-Status > Date: Wed, 30 Oct 2013 18:15:47 GMT > Vary: Accept-Encoding, Brief, Prefer > Preference-Applied: return=minimal, wait=180 > Content-Type: application/xml; charset=utf-8 > Content-Length: 202 > > <?xml version="1.0" encoding="utf-8"?> > <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-356</D:sync-token> > </D:multistatus> > > > > Here is the same sequence of events utilizing streaming with a timeout: > > REPORT /dav/calendars/user/ken/Default/ HTTP/1.1 > Host: localhost > Date: Wed, 30 Oct 2013 18:11:06 GMT > Content-Type: application/xml > Content-Length: 260 > Prefer: return=minimal, wait=180 > Accept: multipart/mixed > > <?xml version="1.0" encoding="UTF-8"?> > <C:sync-collection xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-354</D:sync-token> > <D:sync-level>1</D:sync-level> > <D:prop/> > </C:sync-collection> > > HTTP/1.1 207 Multi-Status > Connection: close > Date: Wed, 30 Oct 2013 18:11:06 GMT > Vary: Accept-Encoding, Brief, Prefer > Preference-Applied: return=minimal, wait=180 > Content-Type: multipart/mixed; > boundary="localhost-29378-1383156666-1025603243" > > This is a message with multiple parts in MIME format. > > --localhost-29378-1383156666-1025603243 > Date: Wed, 30 Oct 2013 18:11:06 GMT > Content-Type: application/xml; charset=utf-8 > Content-Length: 421 > > <?xml version="1.0" encoding="utf-8"?> > <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:response> > <D:href>/dav/calendars/user/ken/Default/4E4B3490-6F01-41B9-AA5B-FE2CD6A30632.ics</D:href> > <D:propstat> > <D:prop/> > <D:status>HTTP/1.1 200 OK</D:status> > </D:propstat> > </D:response> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-355</D:sync-token> > </D:multistatus> > > --localhost-29378-1383156666-1025603243 > Date: Wed, 30 Oct 2013 18:12:47 GMT > Content-Type: application/xml; charset=utf-8 > Content-Length: 375 > > <?xml version="1.0" encoding="utf-8"?> > <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:response> > <D:href>/dav/calendars/user/ken/Default/4E4B3490-6F01-41B9-AA5B-FE2CD6A30632.ics</D:href> > <D:status>HTTP/1.1 404 Not Found</D:status> > </D:response> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-356</D:sync-token> > </D:multistatus> > > --localhost-29378-1383156666-1025603243 > Date: Wed, 30 Oct 2013 18:14:05 GMT > Content-Type: application/xml; charset=utf-8 > Content-Length: 202 > > <?xml version="1.0" encoding="utf-8"?> > <D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> > <D:sync-token>http://cyrusimap.org/ns/sync/1368011844-356</D:sync-token> > </D:multistatus> > > --localhost-29378-1383156666-1025603243-- > > End of MIME multipart body. > > -- > Kenneth Murchison > Principal Systems Software Engineer > Carnegie Mellon University > > > > _______________________________________________ > tc-push-l mailing list > tc-push-l@lists.calconnect.org > http://lists.calconnect.org/mailman/listinfo/tc-push-l >
Received on Thursday, 17 April 2014 09:46:04 UTC