Robots.txt is a powerful tool to control search engine crawling on your blog or website. Let me explain you what this robots.txt does for your blog.
Whenever a search engine bot or crawler comes to your page, robots.txt tells it or guides it which pages of your blog are to be indexed and which ones are not. It can be easily used to specify parts of your website which you don’t want search engines to crawl and index.
For example, you would like to avoid your blog’s login page and plugin pages to be indexed by the search engines. Then you can use robots.txt to achieve this.
How to Create Robots.txt ?
Creating a robots.txt for your blog is a really simple process. Use following steps and you are done.
- Open the text editor or notepad.
- Write the contents in your file( I will tell you in a while).
- Save the file as robots.txt.
- Place it in the root directory of your website such that when you go to : http://mywebsite.com/robots.txt , your robots.txt file should open up in your web browser.
How to Configure Robots.txt ?
Now this is the most important part of this blogging tip. But it’s easy and simple. Let me explain you how to configure your robots.txt.
Each robots.txt is made up of simple records. Each record has two parts:
- User-agent :- This generally is for search engine crawlers or bots.
- Disallow :- This is used to specify directories or pages you don’t want to be indexed by a specific User-agent or bot.
Take a look at following example :-
User-agent: googlebot
Disallow: /images
In the above record, the record for googlebot has been specified. In above example, you are telling googlebot to crawl every page of your blog except those which are in /images directory. Now everything which is in /images will not be crawled and will not be indexed by the googlebot. For example, /images/image1 and /images/image2 will not be crawled.
If you want all search engine bots to be given same crawling rights, then you can use ”*”(without quotes) for the User-agent in your robots.txt. For example,
User-agent: *
Disallow: /images
Above record will exempt your /images directory to be crawled by all search engine spiders and bots.
Thus you saw how easy it was to configure your robots.txt for SEO. What setting are you using for your robots.txt? Do let me know about it.
{ 10 comments… read them below or add one }
Thanks Vineet for your contribution. a while back I written an similar article about complete guide about website meta tag & robots.txt, check it out!
Meta Tag & Robot recently posted..Free Tools For SavVy Internet Marketer
I will definitely read your article Sohel. Thanks.
Vineet Saxena recently posted..The Importance and SEO of Robots.Txt For Your Blog
Hi Vineet….great attempt bro.
I would suggest you should add some more information into that like how to stop browsing different folders (specialy admin and wp-content folders).
Sandeep Kumar recently posted..Keyword Tutorial – How to search Top Keywords and How To use Keywords
Nice post about Robots.txt
One should really master these basic setup steps to have a fully operational and working blog.
Thanks Vineet.
Dukeo recently posted..Answer to All the Emails from Readers
Thanks Dukeo….Robots.txt are an important part of SEO fro your blog.
i'll surely try this on my blog, pls check out my blog http://bloggingpills.com why is my contact not working.. i use fs contact form… any idea?
Hey Gilbert,
I don't understand which contact are you talking about? If you are talking about the contact us page, then I see no problem with that.
Vineet Saxena recently posted..Ways to Increase Your Blog Revenue
Niche post i like it…
I like the way you put your points in blog and explain it explicitly.
thanks
Amazing Blog on seo of robots…!!
Thanks for Sharing!!
Web Design recently posted..Sacha Cosmetics
It was really a great post. I like to suggest all beginner to read Robots.txt form this post. And the way here robots.txt protocols explain was awesome.
thanks
Covetus