W3C home > Mailing lists > Public > www-lib@w3.org > July to September 1999

RE: fast robot

From: Franšois-Louis Mommens <mommens@echo.fr>
Date: Sat, 10 Jul 1999 16:28:57 +0200
To: "Henrik Frystyk Nielsen" <frystyk@w3.org>, <www-lib@w3.org>
Message-ID: <003901becae0$8b66ae80$7e01a8c0@orion.echo-net.net>
I need to stop the transfert even if it close the TCP connection. I can't
download more than a certain size for I won't save the whole document and it
would take bandwidth uselessly.

> -----Message d'origine-----
> De: Henrik Frystyk Nielsen [mailto:frystyk@w3.org]
> Date: vendredi 9 juillet 1999 15:45
> └: Franšois-Louis Mommens; Vladimir Kozlov; www-lib@w3.org
> Objet: RE: fast robot
>
>
> At 15:22 10/07/1999 +0200, Franšois-Louis Mommens wrote:
> >Thanks for your response but it's not exactly what I'm looking for.
> >
> >In fact, I'm working on a search engine project and I only need
> the begining
> >of web pages cause I can't keep the totality.
> >
> >This method of ranges work only with a few amount of web servers. What I
> >realy need is to stop the transfer of data anyway.
>
> That may in fact be much more expensive as this requires that you
> close the
> TCP connection and have to get a new TCP connection up and running. What
> you may want to do is to try and see if the server understands range
> requests and if so then use that, otherwise use HEAD requests.
>
> Henrik
> --
> Henrik Frystyk Nielsen,
> World Wide Web Consortium
> http://www.w3.org/People/Frystyk
>
Received on Friday, 9 July 1999 10:29:43 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 5 February 2014 07:15:16 UTC