How to Stop Search Engines from Crawling Your WordPress Website
Do you want to stop search engines from crawling your WordPress website or from listing it in search results? There are solutions you can use. In this article, we will be looking at the two surefire methods you can use to stop search engines from listing or crawling your website.
Why would you want to stop search engines?
This is probably the question on your mind. It is a valid question considering search engines are the main source of web traffic. One of the main reasons why you would want to stop search engines from indexing or crawling your WordPress website is when you are developing a website live on publicly accessible domain name. In this case you don’t want the search engine to index a website that is in maintenance mode or under construction. Another reason is if yours is a private website or blog. You may also be using your website for internal documents that you don’t want to be accessed publicly.
Even with no links pointing to your website, search engines will still index you. So stopping the search engines from ever crawling or listing your website is the key to remaining private. Here is how you can do this.
Use WordPress built-in feature
WordPress has a feature that allows you to give search engines instructions not to index your website. You just need to go to Settings>>Reading and then check the box that is next to the ‘Search Engine Visibility’ option. Checking this box add this line to the header of your website:
<meta name=’robots’ content=’noindex,follow’/>
WordPress will also modify the robots.txt file to:
User-agent: *
Disallow: /
These lines will tell web crawlers not to index your website. At times the request may not be accepted by all search engines. At times, random images from your website may be indexed. To stop crawling completely, you need to use the following methods.
Method 1: Password protect your website
You achieve this from the cPanel provided by your WordPress hosting provider. You just login to the cPanel dashboard and click on ‘Password Protect Directories’ icon located under ‘Security’ section. Next you select the folder where you installed WordPress. This is normally at public_html folder. If you have several WordPress sites already installed, you need to select the website you wish to password protect. Enter the name of the protected directory then save the changes.
cPanel will reload after saving. The next step is to add your preferred username and password. You are done. Whenever search engine crawlers get to your website, they will be required to enter a username and password to view the site.
Method 2: Password Protecting WordPress with a plugin
For managed WordPress hosting, this is the option for you since you don’t have access to the cPanel. You can use plugins to password protect your site. The two best plugins for this are SeedProd and Password Protected.
You can stop search engines from crawling your WordPress website using one of these methods. Don’t be afraid to ask for expert help from an experienced WordPress web designer if you are ever stuck.