What is WordPress robots.txt file
Today I am going to teach you what is WordPress robots.txt file and why you need one. It is a file which tells Google search engine bot what to show and hide in search results and indexing. There are few misunderstanding related to indexing and non-indexing of content in Robots.txt. I am not going make a long article just follow simple steps, and that’s all.
Why WordPress robots.txt file is relevant
It is an essential part of SEO, and It is important because some time search engine crawls root directory and WordPress admin pages too which is a security risk for any blog. Maybe your blog has the robots.txt file but it is miss-configured, or you are not aware how to optimize it.
Where is the WordPress robots.txt location
Robots.txt file normally located in the root folder of your hosting Cpanel account. You can go to file manager and there you will find your robots.txt file.
WordPress robots.txt disallow
These two lines will allow Google bot to index every page of your blog accept cgi-bin folder which is root directory folder. It means bots will not index cgi-bin folder
WordPress robots.txt allow
Simple if you use ALLOW command defiantly google bot will index this folder, and this is the way how to use ALLOW and DISALLOW commands in the robots.txt file.
Sitemap in WordPress robots.txt
A sitemap is also the most important part of any blog because it has everything like pages, categories, post, etc. In my opinion, Sitemap should be included in the robots.txt file and let the crawler index it too.
Parts of Google Bots
How to create WordPress robots.txt File
Here is an example of my robots.txt file and I hope you can copy it and use it on your blog. All you need to do is create a new text file on your computer using notepad or any text editor and save it as robots.txt. Next, simply upload it to your site’s root folder.
How to check WordPress robots.txt file
To check your robots.txt file simply type http://www.yourdomainname.com/robots.txt in your browser and your robots.txt file will be displayed. Please replace your domain name with your sites name.
Check WordPress robots.txt file in google webmaster tools
This is the last and important point because we are dealing with Google search engine bot so we must go to our google webmaster tools and choose robots.txt from Crawl menu. There you can see tester option submit it and you will see the results.
After this article, I hope you can make and configure your WordPress Robots.txt File and optimize it as per your needs. If you have any questions, please comment below.