abuse from msnbot from 65.54.188.15 (msnbot.msn.com)

msnbot has been doing dumb things on my site, hitting it too
frequently (up to 25+ requests per second) and requesting the
same files over and over again.

It retrieved some files over 1000 times each, though they are
simple static files whose content does not change, with an
explicit expiry time of one day returned in the HTTP headers:

    HEAD http://impressive.net/people/gerald/2003/02/19/14-05-14.rdf
    200 OK
    Cache-Control: max-age=86400
    Connection: close
    Date: Fri, 19 Aug 2005 20:37:57 GMT
    Accept-Ranges: bytes
    ETag: "1cb390-5e2-3e5a65fb"
    Server: Apache/1.3.33 (Debian GNU/Linux) PHP/4.3.10-15 mod_ssl/2.8.22 OpenSSL/0.9.7d
    Content-Length: 1506
    Content-Type: text/xml
    Expires: Sat, 20 Aug 2005 20:37:57 GMT
    Last-Modified: Mon, 24 Feb 2003 18:35:39 GMT

Please fix your bot to be better behaved.

sample log entries:

requesting the same file over and over:

    65.54.188.15 - - [19/Aug/2005:16:26:47 -0400] "GET /people/gerald/2003/02/19/14-05-14.rdf HTTP/1.0" 200 1506 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:16:30:58 -0400] "GET /people/gerald/2003/02/19/14-05-14.rdf HTTP/1.0" 200 1506 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:16:34:17 -0400] "GET /people/gerald/2003/02/19/14-05-14.rdf HTTP/1.0" 200 1506 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:16:36:51 -0400] "GET /people/gerald/2003/02/19/14-05-14.rdf HTTP/1.0" 200 1506 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:16:40:42 -0400] "GET /people/gerald/2003/02/19/14-05-14.rdf HTTP/1.0" 200 1506 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net

(it has requested that URI over 1030 times in the past two days)

hitting my site too frequently:

    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/06/22-08-44.rdf HTTP/1.0" 200 1410 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/03/15-18-29.rdf HTTP/1.0" 200 1421 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/03/16-29-14.rdf HTTP/1.0" 200 1365 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/04/13-56-23.rdf HTTP/1.0" 200 1382 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/04/13-33-49.rdf HTTP/1.0" 200 1399 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/03/16-12-35.rdf HTTP/1.0" 200 1365 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/03/12-58-17.rdf HTTP/1.0" 200 1435 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/14-47-29.rdf HTTP/1.0" 200 1417 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/07-03-20.rdf HTTP/1.0" 200 1409 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/11-42-23.rdf HTTP/1.0" 200 1466 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/12-43-15.rdf HTTP/1.0" 200 1412 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/04/14-57-00.rdf HTTP/1.0" 200 1391 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/14-38-18.rdf HTTP/1.0" 200 1528 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/14-19-00.rdf HTTP/1.0" 200 1414 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2004/09/29/21-02-07.rdf HTTP/1.0" 200 1497 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/04/14-21-51.rdf HTTP/1.0" 200 1402 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/04/12-30-04.rdf HTTP/1.0" 200 1362 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/03/15-41-19.rdf HTTP/1.0" 200 1365 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/11-21-41.rdf HTTP/1.0" 200 1406 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2001/11/08/14-17-37.rdf HTTP/1.0" 200 1414 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net
    65.54.188.15 - - [19/Aug/2005:04:10:38 -0400] "GET /people/gerald/2003/02/19/14-03-46.rdf HTTP/1.0" 200 1443 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" 0 impressive.net

-- 
Gerald Oskoboiny <gerald@impressive.net>
http://impressive.net/people/gerald/

Received on Friday, 19 August 2005 21:00:46 UTC