Robots.txt is a powerful tool to control search engine crawling on your blog or website. Let me explain you what this robots.txt does for your blog.
Whenever a search engine bot or crawler comes to your page, robots.txt tells it or guides it which pages of your blog are to be indexed and which ones are not. It can be easily used to specify parts of your website which you don’t want search engines to crawl and index.
For example, you would like to avoid your blog’s login page and plugin pages to be indexed by the search engines. Then you can use robots.txt to achieve this.
How to Create Robots.txt ?
Creating a robots.txt for your blog is a really simple process. Use following steps and you are done.
- Open the text editor or notepad.
- Write the contents in your file( I will tell you in a while).
- Save the file as robots.txt.
- Place it in the root directory of your website such that when you go to : http://mywebsite.com/robots.txt , your robots.txt file should open up in your web browser.
How to Configure Robots.txt ?
Now this is the most important part of this blogging tip. But it’s easy and simple. Let me explain you how to configure your robots.txt.
Each robots.txt is made up of simple records. Each record has two parts:
- User-agent :- This generally is for search engine crawlers or bots.
- Disallow :- This is used to specify directories or pages you don’t want to be indexed by a specific User-agent or bot.
Take a look at following example :-
In the above record, the record for googlebot has been specified. In above example, you are telling googlebot to crawl every page of your blog except those which are in /images directory. Now everything which is in /images will not be crawled and will not be indexed by the googlebot. For example, /images/image1 and /images/image2 will not be crawled.
If you want all search engine bots to be given same crawling rights, then you can use ”*”(without quotes) for the User-agent in your robots.txt. For example,
Above record will exempt your /images directory to be crawled by all search engine spiders and bots.
Thus you saw how easy it was to configure your robots.txt for SEO. What setting are you using for your robots.txt? Do let me know about it.