Twitter LinkedIn Google+
WP Greet Box icon

Welcome back, visitor!. You might want to subscribe to the RSS feed for online marketing info as Todd posts it.

Switch Reading StyleNighttimeDaytime

MSN Bot Hitting Sites Hard

Online marketing information can change quickly This article is 10 years and 125 days old, and the facts and opinions contained in it may be out of date.

I have neglected to post on this yet, but MSNBot seems to be hitting sites excessively hard. For most of us, this is not a problem, but paying for bandwidth for a spider (BG’s spider no less) is none to appealing to folks with large, high traffic sites.

MSNBot will support a robots.txt file command to slow down crawling if your site is getting “eaten alive”.
Just add:
User-Agent: msnbot
Crawl-Delay: 20

See the full discussion at webmasterworld:
http://www.webmasterworld.com/forum97/93.htm

More information about Todd Malicoat aka stuntdubl.

Twitter LinkedIn Google+ 

Buffer