How To Create WordPress Robots.txt File For Better SEO

wordpress robots.txt file

What is WordPress robots.txt file

Today I am going to teach you what is WordPress robots.txt file and why you need one.  It is a file which tells Google search engine bot what to show and hide in search results and indexing.  There are few misunderstanding related to indexing and non-indexing of content in Robots.txt. I am not going make a long article just follow simple steps, and that’s all.

wordpress robots.txt file example

Why WordPress robots.txt file is relevant

It is an essential part of SEO, and It is important because some time search engine crawls root directory and WordPress admin pages too which is a security risk for any blog. Maybe your blog has the robots.txt file but it is miss-configured, or you are not aware how to optimize it.

Where is the WordPress robots.txt location

Robots.txt file normally located in the root folder of your hosting Cpanel account. You can go to file manager and there you will find your robots.txt file.

wordpress robots.txt Lcation

WordPress robots.txt disallow

User-agent: Googlebot

Disallow: /cgi-bin

These two lines will allow Google bot to index every page of your blog accept cgi-bin folder which is root directory folder. It means bots will not index cgi-bin folder

WordPress robots.txt allow

User-agent: Googlebot

Allow: /cgi-bin

Simple if you use ALLOW command defiantly google bot will index this folder, and this is the way how to use ALLOW and DISALLOW commands in the robots.txt file.

Sitemap in WordPress robots.txt

A sitemap is also the most important part of any blog because it has everything like pages, categories,  post, etc. In my opinion, Sitemap should be included in the robots.txt file and let the crawler index it too.

 wordpress robots.txt sitemap

Parts of Google Bots

 wordpress robots.txt parts

How to create WordPress robots.txt File

Here is an example of my robots.txt file and I hope you can copy it and use it on your blog. All you need to do is create a new text file on your computer using notepad or any text editor and save it as robots.txt. Next, simply upload it to your site’s root folder.

How  to check  WordPress robots.txt file

To check your robots.txt file simply type http://www.yourdomainname.com/robots.txt   in your browser and your robots.txt file will be displayed.  Please replace your domain name with your sites name.

wordpress robots.txt check

Check WordPress robots.txt file in google webmaster tools

This is the last and important point because we are dealing with Google search engine bot so we must go to our google webmaster tools and choose robots.txt from Crawl menu. There you can see tester option submit it and you will see the results.

wordpress robots.txt webmaster tool

wordpress robots.txt test

After this article, I hope you can make and configure your WordPress Robots.txt File and optimize it as per your needs. If you have any questions, please comment below.