Today I’ll show you how to use the file robots.txt.
But first, let’s get why, in fact, we need the file robots.txt.
The robots.txt file is to specify the search of search engines which files and folders you do not want to be indexed. It should be located in the root directory of a site.
Very small html-sites, which contain 10-30 pages do not need a robots.txt file, as all the files of this site should be indexed.
In the case of large dynamic websites there is a set of special pages, it is not intended for publication, and to update the site requires a comfortable internal structure of the site and the active use of the file robots.txt.
The robots.txt file usually looks like this:
User-agent: *
Disallow: / delo.php
Disallow: / d123 /
Disallow: / travel / dat /
Line User-agent: specifies a particular robot or * – all robots. In the Disallow line the path to the folder or file specifically prohibited for indexing (the path is absolute, measured from the root site). To allow the robot to access some parts of the site or the entire site, use the directive Allow. Blank lines in between the User-agent string and Disallow, Allow should not be.
If you use the site map described with the format sitemaps.xml, and you want the robot to learn about it, specify the path to sitemaps.xml, as a parameter to the directive Sitemap (if multiple files, select all)
If your site has a mirror, a special robot mirror identifies them and forms a group of mirrors of your site. The search would include only the primary mirror. To do this you must specify the robots.txt, use the directive Host, defining it as a parameter of the name of the mirror. Directive Host, as search engine treats Help, does not guarantee a specified range of the primary mirror, however, the algorithm takes into account when deciding its high priority.
You can also specify the time the robot visits the page of your site, you can use this directive Crawl-delay. It allows you to specify a search robot minimum time (in seconds) between the end of injection of a single page and the beginning of the next injection. It is imperative to ensure compatibility with the robots, Crawl-delay directive should be added to the group, starting with the entry “User-Agent”, immediately after the directive Disallow (Allow).
The search engine supports fractional values Crawl-Delay, for example, 0.5. This does not guarantee that the crawler will go to your site every half second, but gives the robot more freedom and allows you to speed up the bypass site.
As you can see everything is easy and clear. Use it on you sites, thus it can help the search engine to correctly read your webpage.
Interested in making money online without investment? You are invited to check out this make money online site. It is the very place on the Internet where you can get info on how to get additional income and arrange everything for it.
In addition, I would like to give some general tips. Today the online technologies give you a truly unique chance to choose what you need at the best terms which are available on the market. Funny, but most of the people don’t use this opportunity. In real life it means that you should use all the tools of today to get the info that you need.
One more thing. The topic has become very popular recently. So search Google and other search engines. Check out social networks and check topics which are related to yours. Go to the niche forums and participate in the discussion. All this will help you to create a concept of such kind of work.
And also I would advise you to sign up for the RSS feed on this blog as we will do everything possible to keep updating this blog with new info about making money online without investment and other important issues.