RE: Multi-server HTTP

Hi Mark, all,

Thanks for your response. Unfortunately we have not received any further
feedback on this, which is a same since we'd really like to know if
there is interest in trying to move this forward.

I have (admittedly only briefly) looked at metalink. It seems to cover
some of what we need (list of mirrors, pieces, checksumming) but seems
mostly to be concerned with finding a single appropriate source rather
than downloading from multiple HTTP servers. This seems to mostly be a
client rather than a spec choice, however. Nevertheless, one of the
disadvantages of metalink, from our point of view, is that it is an
overhead. This is negligible for large files, but one of our (longer
term) use cases is for mirrors of a whole site allowing e.g. a set of
images to be downloaded from different servers. As such, there is a
moderate delay before a download would start since first the metalink
must be downloaded, then decisions made, then new downloads started.

In our case, the download starts immediately, just as in standard HTTP,
and the client can take over the requesting of various parts when it is
ready, so there is no delay introduced by metadata handshaking.

Our solution is indeed designed to operate on the same URLs as  It seems
that it is feasible for metalink to also be done transparently (by the
client declaring "Accept: application/metalink+xml" as I understand it).

So, folks... any more thoughts? :)

Regards,
Alan

> -----Original Message-----
> From: Mark Nottingham [mailto:mnot@mnot.net]
> Sent: 25 August 2009 07:14
> To: Ford, Alan
> Cc: ietf-http-wg@w3.org; Mark Handley
> Subject: Re: Multi-server HTTP
> 
> Alan and Mark,
> 
> There unfortunately hasn't been much discussion of this yet, at least
> on the list. Has there been progress elsewhere?
> 
> For my part, this looks like interesting work. If I understand it
> correctly, it's entirely application-layer (or at least able to be
> implemented within the application layer), so if you want to, I think
> it's entirely appropriate to discuss it on this list.
> 
> Also, have you made contact with the folks doing Metalink
> <http://www.metalinker.org/
>  >? They have deployed implementations, and it's my understanding that
> they're looking at revising the spec now, so it may an excellent time
> to collaborate.
> 
> Personally, I'd like to see the end result able to use the same URL
> for multi-server downloads and "traditional" single-server downloads;
> i.e., it should be transparent to clients.
> 
> Cheers,
> 
> 
> On 31/07/2009, at 9:59 PM, Ford, Alan wrote:
> 
> > Hi all,
> >
> > At the IETF this week, Mark Handley and I submitted a
floating-an-idea
> > draft on multi-server HTTP and presented it in tsvarea.
> >
> > http://www.ietf.org/id/draft-ford-http-multi-server-00.txt
> >
> > Slides are at:
http://www.ietf.org/proceedings/75/slides/tsvarea-0.pdf
> >
> > I realise Transport Area didn't capture a large number of HTTP
> > people -
> > the main reason for presenting it there was our key motivation was
to
> > improve Internet resource usage, and we have been doing other such
> > work
> > (notably multipath TCP) in that area. We were also very short on
> > preparation time before the IETF - so apologies for missing many of
> > you
> > guys.
> >
> > However, we would very much like input and guidance from the HTTP
> > community. I am grateful to Henrik Nordstrom for suggesting we
should
> > bring it to the HTTPbis WG, even though as an extension it is not
> > within
> > the charter.
> >
> > This is a brief summary of the proposal:
> >
> >  * We are aiming to achieve better usage of Internet resources by
> > applying BitTorrent-like chunked downloading of large files from
> > different servers.
> >  * Upon connection to a Multi-Server HTTP server, when a client says
> > they are Multi-server capable, in the response the server will
> > provide a
> > list of mirrors for that resource, a checksum for the file, and a
> > chunk
> > of the file with a Content-Range header.
> >  * The client will then send more GET requests, this time with
Range:
> > headers, to the original server and to zero or more of the mirror
> > servers, along with a verification header to ensure the checksum
> > matches
> > and so the resource is the same. The client will handle the
scheduling
> > of Range requests in order to make the most effective use of the
least
> > loaded servers.
> >
> > We realise that the draft itself is not making the best use of
> > existing
> > proposals. During the presentation, Instance-Digests (RFC3230) were
> > mentioned which look ideal instead of X-Checksum, although we will
> > still
> > need an If-Digest-Match header. Content-MD5 was also suggested but
> > that
> > appears to be a checksum of just the data that is sent, not the
whole
> > resource.
> >
> > I discounted ETags along with If-Match in the proposal since RFC2616
> > says "Entity tags are used for comparing two or more entities from
the
> > same requested resource" but if I have understood the terminology
> > correctly, in our proposal we are fetching chunks from different
> > resources (even though the content should be the same). Indeed the
RFC
> > also says, "The use of the same entity tag value in conjunction with
> > entities obtained by requests on different URIs does not imply the
> > equivalence of those entities." Please correct me if I'm wrong!
> >
> > There is also a question of whether we could make further
extensions,
> > specifically:
> >
> >  * Wildcarded mirror lists (e.g. a server that mirrors all
/a/*.jpg).
> >  * Checksums could be provided for file chunks allowing broken
chunks
> > to be re-fetched.
> >  * Servers could store multiple versions of the file indexed by
> > checksum.
> >  * Initial servers could send no, or very little, data itself, and
> > purely act as a load balancer; or redirect immediately when it's
> > overloaded.
> >
> > These may change the mechanism quite considerably, however (e.g.
with
> > wildcards, no longer would you be getting all checksums from the
same
> > server; and for verification checksum chunks need to be
pre-determined
> > and calculated).
> >
> > We believe that the extension as it stands can bring significant
> > benefit
> > to HTTP, making much more efficient use of Internet resources.
> > Experiments have been conducted that suggest it has no negative
impact
> > in every scenario in which it was tested.
> >
> > Looking forward to your comments and advice!
> >
> > Regards,
> > Alan
> >
> >
------------------------------------------------------------------------
> > Alan Ford
> >
> > Tel:	+44 (0)1794 833465
> > Fax:	+44 (0)1794 833433
> > alan.ford@roke.co.uk
> >
> >
> > --
> > Roke Manor Research Ltd, Romsey,
> > Hampshire, SO51 0ZN, United Kingdom
> >
> > A Siemens company
> > Registered in England & Wales at:
> > Siemens plc, Faraday House, Sir William Siemens Square,
> > Frimley, Camberley, GU16 8QD. Registered No: 267550
> >
------------------------------------------------------------------------
> > Visit our website at www.roke.co.uk
> >
------------------------------------------------------------------------
> > The information contained in this e-mail and any attachments is
> > proprietary to Roke Manor Research Ltd and must not be passed to any
> > third party without permission. This communication is for
information
> > only and shall not create or change any contractual relationship.
> >
------------------------------------------------------------------------
> >
> > Please consider the environment before printing this email
> >
> >
> 
> 
> --
> Mark Nottingham     http://www.mnot.net/


-- 
Roke Manor Research Ltd, Romsey,
Hampshire, SO51 0ZN, United Kingdom

A Siemens company
Registered in England & Wales at:
Siemens plc, Faraday House, Sir William Siemens Square,
Frimley, Camberley, GU16 8QD. Registered No: 267550
------------------------------------------------------------------------
Visit our website at www.roke.co.uk
------------------------------------------------------------------------
The information contained in this e-mail and any attachments is
proprietary to Roke Manor Research Ltd and must not be passed to any
third party without permission. This communication is for information
only and shall not create or change any contractual relationship.
------------------------------------------------------------------------

Please consider the environment before printing this email

Received on Tuesday, 25 August 2009 09:25:46 UTC