RE: fast robot

I need to stop the transfert even if it close the TCP connection. I can't
download more than a certain size for I won't save the whole document and it
would take bandwidth uselessly.

> -----Message d'origine-----
> De: Henrik Frystyk Nielsen [mailto:frystyk@w3.org]
> Date: vendredi 9 juillet 1999 15:45
> À: François-Louis Mommens; Vladimir Kozlov; www-lib@w3.org
> Objet: RE: fast robot
>
>
> At 15:22 10/07/1999 +0200, François-Louis Mommens wrote:
> >Thanks for your response but it's not exactly what I'm looking for.
> >
> >In fact, I'm working on a search engine project and I only need
> the begining
> >of web pages cause I can't keep the totality.
> >
> >This method of ranges work only with a few amount of web servers. What I
> >realy need is to stop the transfer of data anyway.
>
> That may in fact be much more expensive as this requires that you
> close the
> TCP connection and have to get a new TCP connection up and running. What
> you may want to do is to try and see if the server understands range
> requests and if so then use that, otherwise use HEAD requests.
>
> Henrik
> --
> Henrik Frystyk Nielsen,
> World Wide Web Consortium
> http://www.w3.org/People/Frystyk
>

Received on Friday, 9 July 1999 10:29:43 UTC