Google has complicated
algorithms that determine how much to crawl each site. Our goal is to crawl as
many pages from your site as we can on each visit without overwhelming your
server's bandwidth.
If you think
Google is crawling your site too often and slowdowns your server, you can change the crawl rate means the
time used by Googlebot to crawl your website for sites that are at the root
level.
Take an example www.example.com and http://subdomain.example.com.
Changing the
crawl rate can cause some problems Google will not be able to crawl at a faster
rate than the custom rate you set. Always do research when to change the crawl
rate. So don't do this unless you are
noticing detailed problems caused by Googlebot accessing your servers too
often.
And also you can’t
change the crawl rate for sites that are not at the root level. For example the
root like this www.example.com/folder
Let see how to change
the crawl rate of the website.
- Login to your webmasters tool account.
- In the webmasters tools home page, select your website you want to change the crawl rate.
- Under configuration, select settings
- In the crawl rate section, there are two sections for crawl rate.
Let google optimize for my site.
Limit Google’s maximum crawl rate
When you select
the second option you can change your crawl rate of your choice.
Source from
Google Support
No comments:
Post a Comment