0 Comments
MSN Bot Hitting Sites Hard
Welcome back! Good to see you. If you haven't seen it, here's the archive of my best writing. Thanks for visiting!
I microblog now more than I blog at Twitter.com/stuntdubl
Feel free to drop me a line sometime or subscribe in case I ever post again.
I have neglected to post on this yet, but MSNBot seems to be hitting sites excessively hard. For most of us, this is not a problem, but paying for bandwidth for a spider (BG’s spider no less) is none to appealing to folks with large, high traffic sites.
MSNBot will support a robots.txt file command to slow down crawling if your site is getting “eaten alive”.
Just add:
User-Agent: msnbot
Crawl-Delay: 20
See the full discussion at webmasterworld:
http://www.webmasterworld.com/forum97/93.htm
