RE: fast robot

At 15:22 10/07/1999 +0200, François-Louis Mommens wrote:
>Thanks for your response but it's not exactly what I'm looking for.
>
>In fact, I'm working on a search engine project and I only need the begining
>of web pages cause I can't keep the totality.
>
>This method of ranges work only with a few amount of web servers. What I
>realy need is to stop the transfer of data anyway.

That may in fact be much more expensive as this requires that you close the
TCP connection and have to get a new TCP connection up and running. What
you may want to do is to try and see if the server understands range
requests and if so then use that, otherwise use HEAD requests.

Henrik
--
Henrik Frystyk Nielsen,
World Wide Web Consortium
http://www.w3.org/People/Frystyk

Received on Friday, 9 July 1999 09:45:58 UTC