W3C home > Mailing lists > Public > www-lib@w3.org > July to September 1999

RE: fast robot

From: Henrik Frystyk Nielsen <frystyk@w3.org>
Date: Fri, 09 Jul 1999 09:45:28 -0400
Message-Id: <3.0.5.32.19990709094528.02cf5ac0@localhost>
To: "François-Louis Mommens" <mommens@echo.fr>, "Vladimir Kozlov" <Vladimir.Kozlov@idm.ru>, <www-lib@w3.org>
At 15:22 10/07/1999 +0200, François-Louis Mommens wrote:
>Thanks for your response but it's not exactly what I'm looking for.
>
>In fact, I'm working on a search engine project and I only need the begining
>of web pages cause I can't keep the totality.
>
>This method of ranges work only with a few amount of web servers. What I
>realy need is to stop the transfer of data anyway.

That may in fact be much more expensive as this requires that you close the
TCP connection and have to get a new TCP connection up and running. What
you may want to do is to try and see if the server understands range
requests and if so then use that, otherwise use HEAD requests.

Henrik
--
Henrik Frystyk Nielsen,
World Wide Web Consortium
http://www.w3.org/People/Frystyk
Received on Friday, 9 July 1999 09:45:58 UTC

This archive was generated by hypermail 2.3.1 : Wednesday, 5 February 2014 07:15:16 UTC