Update for SEO Experts

New script is introduced to control crawler based search engines. Through this script you can notify search engines how much time crawler should crawl any particular site.

crawl-delay: 10

You can write the above script in robots.txt file. Many seo experts are using this script to optimize websites. But google is not following this script. It can be good for other crawler based search engines. It may effect your rank in google search. Google is following different algorythm. If you want to use this script then it is recommended to use it for other crawler based search engines like Yahoo, bing etc.