MSN Bot Hitting Sites Hard
I have neglected to post on this yet, but MSNBot seems to be hitting sites excessively hard. For most of us, this is not a problem, but paying for bandwidth for a spider (BG’s spider no less) is none to appealing to folks with large, high traffic sites.
MSNBot will support a robots.txt file command to slow down crawling if your site is getting “eaten alive”.
Just add:
User-Agent: msnbot
Crawl-Delay: 20
See the full discussion at webmasterworld:
http://www.webmasterworld.com/forum97/93.htm
Leave a Reply