Courting the crawl
Your first important step in courting the crawl is learning how actually Googlebot works
How often do Search Engines Crawl?
Search engines constantly crawl the web. depend on how often you update the content.
What Googlebot does on your site?
Once Googlebot is on your site, it crawls each page in turn. and then it ...
Controlling Robots or bot
To prevent Google from crawling certain pages, the best method is to use robots.txt to control robots