advertise with

The Crawler

The first part of a search engine is the crawler (AKA spider, robot, and bot). The crawler reads pages into the search engine raw database of every page the search engine crawler has read. Some important things to know about crawlers…

  1. Search bots periodically reread every web page in their database.
    • You don’t have to do anything when you change your site. If your site is already listed in a search engine and you make changes to your website, search engines will eventually update their information.
    • You do have to be patient. The frequency of re-crawling varies depending on the search engine, the importance of your site (as determined by the search engines), and how often you update it. It could take a day or it could take more than a week.
  2. Search bots follow links on the pages that have already been crawled in order to find new pages.
    • If you add a new page to your website or create a new website, it’s important to add a link to it on a web page that is already in the search engines.
    • If a web page that is already in the search engines has a link to your new page or new website, you do not need to submit a request to the search engines to crawl the new page or website, though it may still be worth doing just to be sure your website will be found quickly.
  3. Some search engines have forms you can submit to request a website be crawled.
    • If you have a new website and no sites link to it, search bots will not be able to find it. In this case, submitting a form to the search engine requesting your site be listed or indexed can get it into search engines.
    • Because human-powered directories do not have search bots/crawlers, to be listed in them you must submit a request form.
Related SEO Tips

How Google finds sites and pages?
more knowledge about how google can finds your site and page

Page Rank Explained (Part 3)
Here are page rank explaination part three, tell us about how to increase page rank.

Search Engine Ranking
Search engines are careful the most important source of traffic for any website.