RSS Feed of this site
RSS Feed of this Site

Enter your address for e-mailed updates

say hey

Stuntdubl Internet Marketing Consulting
Don't try this at home!

Saturday, June 19, 2004

MSN Bot Hitting Sites Hard

I have neglected to post on this yet, but MSNBot seems to be hitting sites excessively hard. For most of us, this is not a problem, but paying for bandwidth for a spider (BG's spider no less) is none to appealing to folks with large, high traffic sites.

MSNBot will support a robots.txt file command to slow down crawling if your site is getting "eaten alive".
Just add:
User-Agent: msnbot
Crawl-Delay: 20

See the full discussion at webmasterworld:
http://www.webmasterworld.com/forum97/93.htm



Older Posts:
  • MarketingVOX is Cool. Spyware and Government Legislation is Not.
  • Don't Get Scammed When Buying Links
  • Google Invests in Overseas Search Engine Baidu
  • 101 Free Search Engine Optimization Tips
  • Keyword Arbitrage - Yahoo Advertising on Google
  • Get Into Google News? Thread at WebmasterWorld.com
  • Search Engine Milestones for May 2004
  • Blogging about Blogging about Business Blogs
  • $2.3 Billion in Internt Advertising - New Quarterly Record!
  • Everything That is Wrong With America

  • Powered by Blogger

    ©Copyright 2004 - Infinity - Stuntdubl Internet Marketing
    Please don't steal stuff from anyone. Karma.

    Creative Commons License

    This work is licensed under a Creative Commons License.