Re: Robots & Persistent connexion, pipelining

>>> Mihai Preda <Mihai.Preda@inria.fr> 22-Aug-00 1:05:14 PM >>>

>But, if I want to get 10 pages from a server, wouldn't it
>be more advantageous (for the server too) to get the 10pages 
>at once (with persistent c.) and afterwards leave it in peace for 
>300sec(30*10) rather than open a TCP con. and get a page 
>every 30sec, 10 times?

Well... there is still the question of processor and disk i/o util
but...


>First, I'd like to know what is the consensus in this matter.

I think persistent connections are all right for robots. 

Trouble is you can't gaurantee you're going to get one can you?


>Second, I'd like to know what do you think about this proposition,
> which is aimed to allow the use of persistent c. by robots:
>So, we are againt server-overload, and we allocate 30sec/page. 
>But we allow to get a limitted number of pages (say, 5 or 10)
together 
>through a persistent con., with the condition to leave the server
rest 
>longer afterwards. If we get 5 pages at once, we won't ask anything

>from the same server for the next 30*5 seconds. What do you think?

I don't think that's necessary myself... if you get a persistent
connection you should be able to get 5-10 pages and then drop for 30
seconds.

Still... it may be worth doing... it would take a long time for rule
changes to percolate through to the real net (as it were).


Nic

Received on Tuesday, 22 August 2000 08:32:25 UTC