Business Management Consultant - Stuntdubl Search and Marketing Consulting

If you're new here, you may want to learn more about me, introduce yourself or check out my best posts. Thanks for visiting!

I have neglected to post on this yet, but MSNBot seems to be hitting sites excessively hard. For most of us, this is not a problem, but paying for bandwidth for a spider (BG’s spider no less) is none to appealing to folks with large, high traffic sites.

MSNBot will support a robots.txt file command to slow down crawling if your site is getting “eaten alive”.
Just add:
User-Agent: msnbot
Crawl-Delay: 20

See the full discussion at webmasterworld:
http://www.webmasterworld.com/forum97/93.htm