RE: Multi-server HTTP

Well that would certainly be a great solution for many use cases. If
there was a distributed set of virtual hosts of a given server, I can
see that working quite well. Plus, this would solve the ETag issue that
I mention (assuming I have understood correctly - nobody has yet
corrected me!), since in this case the client /is/ requesting the same
resource from each IP address.

There are two extra issues that spring to mind that our solution
handles, however, that this does not:

It permits the use of any mirroring service (e.g. mirror.ac.uk,
sourceforge) where a mirror of the server is not a dedicated virtual
host, but is a subdirectory of a larger mirror, or indeed is just a
completely different server.

Also, in the case you outline, how would the client know what Range: to
request to begin with, since it does not know the size of the resource
at that stage? I realise our proposal is not hugely elegant in that
sense either, but it seemed the best way to get the connection going (by
initially delegating a range choice to the server, which of course knows
the resource size).

Regards,
Alan

> -----Original Message-----
> From: Robert Siemer [mailto:Robert.Siemer-http@backsla.sh]
> Sent: 25 August 2009 07:53
> To: Mark Nottingham
> Cc: Ford, Alan; ietf-http-wg@w3.org; Mark Handley
> Subject: Re: Multi-server HTTP
> 
> It there anything big different from setting up a bunch of servers
> behind a DNS round-robin and changing the HTTP-client to go for all
IPs?
> 
> The client could wait for the response headers from the first server
and
> spawn more connections with range requests using ETag and URL.
> 
> Basically everything in HTTP already, apart from some
clarifications...
> 
> 
> Robert
> 
> On Tue, 2009-08-25 at 16:13 +1000, Mark Nottingham wrote:
> > Alan and Mark,
> >
> > There unfortunately hasn't been much discussion of this yet, at
least
> > on the list. Has there been progress elsewhere?
> >
> > For my part, this looks like interesting work. If I understand it
> > correctly, it's entirely application-layer (or at least able to be
> > implemented within the application layer), so if you want to, I
think
> > it's entirely appropriate to discuss it on this list.
> >
> > Also, have you made contact with the folks doing Metalink
> <http://www.metalinker.org/
> >  >? They have deployed implementations, and it's my understanding
that
> > they're looking at revising the spec now, so it may an excellent
time
> > to collaborate.
> >
> > Personally, I'd like to see the end result able to use the same URL
> > for multi-server downloads and "traditional" single-server
downloads;
> > i.e., it should be transparent to clients.
> >
> > Cheers,
> >
> >
> > On 31/07/2009, at 9:59 PM, Ford, Alan wrote:
> >
> > > Hi all,
> > >
> > > At the IETF this week, Mark Handley and I submitted a
floating-an-idea
> > > draft on multi-server HTTP and presented it in tsvarea.
> > >
> > > http://www.ietf.org/id/draft-ford-http-multi-server-00.txt
> > >
> > > Slides are at:
http://www.ietf.org/proceedings/75/slides/tsvarea-0.pdf
> > >
> > > I realise Transport Area didn't capture a large number of HTTP
> > > people -
> > > the main reason for presenting it there was our key motivation was
to
> > > improve Internet resource usage, and we have been doing other such
> > > work
> > > (notably multipath TCP) in that area. We were also very short on
> > > preparation time before the IETF - so apologies for missing many
of
> > > you
> > > guys.
> > >
> > > However, we would very much like input and guidance from the HTTP
> > > community. I am grateful to Henrik Nordstrom for suggesting we
should
> > > bring it to the HTTPbis WG, even though as an extension it is not
> > > within
> > > the charter.
> > >
> > > This is a brief summary of the proposal:
> > >
> > >  * We are aiming to achieve better usage of Internet resources by
> > > applying BitTorrent-like chunked downloading of large files from
> > > different servers.
> > >  * Upon connection to a Multi-Server HTTP server, when a client
says
> > > they are Multi-server capable, in the response the server will
> > > provide a
> > > list of mirrors for that resource, a checksum for the file, and a
> > > chunk
> > > of the file with a Content-Range header.
> > >  * The client will then send more GET requests, this time with
Range:
> > > headers, to the original server and to zero or more of the mirror
> > > servers, along with a verification header to ensure the checksum
> > > matches
> > > and so the resource is the same. The client will handle the
scheduling
> > > of Range requests in order to make the most effective use of the
least
> > > loaded servers.
> > >
> > > We realise that the draft itself is not making the best use of
> > > existing
> > > proposals. During the presentation, Instance-Digests (RFC3230)
were
> > > mentioned which look ideal instead of X-Checksum, although we will
> > > still
> > > need an If-Digest-Match header. Content-MD5 was also suggested but
> > > that
> > > appears to be a checksum of just the data that is sent, not the
whole
> > > resource.
> > >
> > > I discounted ETags along with If-Match in the proposal since
RFC2616
> > > says "Entity tags are used for comparing two or more entities from
the
> > > same requested resource" but if I have understood the terminology
> > > correctly, in our proposal we are fetching chunks from different
> > > resources (even though the content should be the same). Indeed the
RFC
> > > also says, "The use of the same entity tag value in conjunction
with
> > > entities obtained by requests on different URIs does not imply the
> > > equivalence of those entities." Please correct me if I'm wrong!
> > >
> > > There is also a question of whether we could make further
extensions,
> > > specifically:
> > >
> > >  * Wildcarded mirror lists (e.g. a server that mirrors all
/a/*.jpg).
> > >  * Checksums could be provided for file chunks allowing broken
chunks
> > > to be re-fetched.
> > >  * Servers could store multiple versions of the file indexed by
> > > checksum.
> > >  * Initial servers could send no, or very little, data itself, and
> > > purely act as a load balancer; or redirect immediately when it's
> > > overloaded.
> > >
> > > These may change the mechanism quite considerably, however (e.g.
with
> > > wildcards, no longer would you be getting all checksums from the
same
> > > server; and for verification checksum chunks need to be
pre-determined
> > > and calculated).
> > >
> > > We believe that the extension as it stands can bring significant
> > > benefit
> > > to HTTP, making much more efficient use of Internet resources.
> > > Experiments have been conducted that suggest it has no negative
impact
> > > in every scenario in which it was tested.
> > >
> > > Looking forward to your comments and advice!
> > >
> > > Regards,
> > > Alan
> > >
> > >
------------------------------------------------------------------------
> > > Alan Ford
> > >
> > > Tel:	+44 (0)1794 833465
> > > Fax:	+44 (0)1794 833433
> > > alan.ford@roke.co.uk
> > >
> > >
> > > --
> > > Roke Manor Research Ltd, Romsey,
> > > Hampshire, SO51 0ZN, United Kingdom
> > >
> > > A Siemens company
> > > Registered in England & Wales at:
> > > Siemens plc, Faraday House, Sir William Siemens Square,
> > > Frimley, Camberley, GU16 8QD. Registered No: 267550
> > >
------------------------------------------------------------------------
> > > Visit our website at www.roke.co.uk
> > >
------------------------------------------------------------------------
> > > The information contained in this e-mail and any attachments is
> > > proprietary to Roke Manor Research Ltd and must not be passed to
any
> > > third party without permission. This communication is for
information
> > > only and shall not create or change any contractual relationship.
> > >
------------------------------------------------------------------------
> > >
> > > Please consider the environment before printing this email
> > >
> > >
> >
> >
> > --
> > Mark Nottingham     http://www.mnot.net/
> >
> >
> 


-- 
Roke Manor Research Ltd, Romsey,
Hampshire, SO51 0ZN, United Kingdom

A Siemens company
Registered in England & Wales at:
Siemens plc, Faraday House, Sir William Siemens Square,
Frimley, Camberley, GU16 8QD. Registered No: 267550
------------------------------------------------------------------------
Visit our website at www.roke.co.uk
------------------------------------------------------------------------
The information contained in this e-mail and any attachments is
proprietary to Roke Manor Research Ltd and must not be passed to any
third party without permission. This communication is for information
only and shall not create or change any contractual relationship.
------------------------------------------------------------------------

Please consider the environment before printing this email

Received on Tuesday, 25 August 2009 09:25:47 UTC