Friday, October 26, 2012

Increase or Decrease your Google crawl rate for your website


Google has complicated algorithms that determine how much to crawl each site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.
If you think Google is crawling your site too often and slowdowns your server, you can change the crawl rate means the time used by Googlebot to crawl your website for sites that are at the root level. 

Take an example www.example.com and http://subdomain.example.com

Changing the crawl rate can cause some problems Google will not be able to crawl at a faster rate than the custom rate you set. Always do research when to change the crawl rate.  So don't do this unless you are noticing detailed problems caused by Googlebot accessing your servers too often.

And also you can’t change the crawl rate for sites that are not at the root level. For example the root like this  www.example.com/folder 

 Let see how to change the crawl rate of the website.

  • Login to your webmasters tool account.
  • In the webmasters tools home page, select your website you want to change the crawl rate.
  • Under configuration, select settings
  • In the crawl rate section, there are two sections for crawl rate.
Let google optimize for my site.
Limit Google’s maximum crawl rate

When you select the second option you can change your crawl rate of your choice. 

Source from Google Support

No comments:

Post a Comment