Re: Multi-server HTTP

Alan and Mark,

There unfortunately hasn't been much discussion of this yet, at least  
on the list. Has there been progress elsewhere?

For my part, this looks like interesting work. If I understand it  
correctly, it's entirely application-layer (or at least able to be  
implemented within the application layer), so if you want to, I think  
it's entirely appropriate to discuss it on this list.

Also, have you made contact with the folks doing Metalink <http://www.metalinker.org/ 
 >? They have deployed implementations, and it's my understanding that  
they're looking at revising the spec now, so it may an excellent time  
to collaborate.

Personally, I'd like to see the end result able to use the same URL  
for multi-server downloads and "traditional" single-server downloads;  
i.e., it should be transparent to clients.

Cheers,


On 31/07/2009, at 9:59 PM, Ford, Alan wrote:

> Hi all,
>
> At the IETF this week, Mark Handley and I submitted a floating-an-idea
> draft on multi-server HTTP and presented it in tsvarea.
>
> http://www.ietf.org/id/draft-ford-http-multi-server-00.txt
>
> Slides are at: http://www.ietf.org/proceedings/75/slides/tsvarea-0.pdf
>
> I realise Transport Area didn't capture a large number of HTTP  
> people -
> the main reason for presenting it there was our key motivation was to
> improve Internet resource usage, and we have been doing other such  
> work
> (notably multipath TCP) in that area. We were also very short on
> preparation time before the IETF - so apologies for missing many of  
> you
> guys.
>
> However, we would very much like input and guidance from the HTTP
> community. I am grateful to Henrik Nordstrom for suggesting we should
> bring it to the HTTPbis WG, even though as an extension it is not  
> within
> the charter.
>
> This is a brief summary of the proposal:
>
>  * We are aiming to achieve better usage of Internet resources by
> applying BitTorrent-like chunked downloading of large files from
> different servers.
>  * Upon connection to a Multi-Server HTTP server, when a client says
> they are Multi-server capable, in the response the server will  
> provide a
> list of mirrors for that resource, a checksum for the file, and a  
> chunk
> of the file with a Content-Range header.
>  * The client will then send more GET requests, this time with Range:
> headers, to the original server and to zero or more of the mirror
> servers, along with a verification header to ensure the checksum  
> matches
> and so the resource is the same. The client will handle the scheduling
> of Range requests in order to make the most effective use of the least
> loaded servers.
>
> We realise that the draft itself is not making the best use of  
> existing
> proposals. During the presentation, Instance-Digests (RFC3230) were
> mentioned which look ideal instead of X-Checksum, although we will  
> still
> need an If-Digest-Match header. Content-MD5 was also suggested but  
> that
> appears to be a checksum of just the data that is sent, not the whole
> resource.
>
> I discounted ETags along with If-Match in the proposal since RFC2616
> says "Entity tags are used for comparing two or more entities from the
> same requested resource" but if I have understood the terminology
> correctly, in our proposal we are fetching chunks from different
> resources (even though the content should be the same). Indeed the RFC
> also says, "The use of the same entity tag value in conjunction with
> entities obtained by requests on different URIs does not imply the
> equivalence of those entities." Please correct me if I'm wrong!
>
> There is also a question of whether we could make further extensions,
> specifically:
>
>  * Wildcarded mirror lists (e.g. a server that mirrors all /a/*.jpg).
>  * Checksums could be provided for file chunks allowing broken chunks
> to be re-fetched.
>  * Servers could store multiple versions of the file indexed by
> checksum.
>  * Initial servers could send no, or very little, data itself, and
> purely act as a load balancer; or redirect immediately when it's
> overloaded.
>
> These may change the mechanism quite considerably, however (e.g. with
> wildcards, no longer would you be getting all checksums from the same
> server; and for verification checksum chunks need to be pre-determined
> and calculated).
>
> We believe that the extension as it stands can bring significant  
> benefit
> to HTTP, making much more efficient use of Internet resources.
> Experiments have been conducted that suggest it has no negative impact
> in every scenario in which it was tested.
>
> Looking forward to your comments and advice!
>
> Regards,
> Alan
>
> ------------------------------------------------------------------------
> Alan Ford
>
> Tel:	+44 (0)1794 833465
> Fax:	+44 (0)1794 833433
> alan.ford@roke.co.uk
>
>
> -- 
> Roke Manor Research Ltd, Romsey,
> Hampshire, SO51 0ZN, United Kingdom
>
> A Siemens company
> Registered in England & Wales at:
> Siemens plc, Faraday House, Sir William Siemens Square,
> Frimley, Camberley, GU16 8QD. Registered No: 267550
> ------------------------------------------------------------------------
> Visit our website at www.roke.co.uk
> ------------------------------------------------------------------------
> The information contained in this e-mail and any attachments is
> proprietary to Roke Manor Research Ltd and must not be passed to any
> third party without permission. This communication is for information
> only and shall not create or change any contractual relationship.
> ------------------------------------------------------------------------
>
> Please consider the environment before printing this email
>
>


--
Mark Nottingham     http://www.mnot.net/

Received on Tuesday, 25 August 2009 06:14:23 UTC