How to Enable Custom Robots.txt file in Blogger and fix indexing problem?

Are you facing an indexing problem and want to enable robots.txt on your Blogger website then this article is definitely helpful for you.

Here, I have explained what is a robots.txt file and why it is important for SEO. I will also tell you how you can set up a robots.txt file on your Blogger website and index your article faster.

What is a Robots.txt file?

Robots.txt file tells search engine crawlers or bots about which URLs they can access and crawl to index them on their database.

This is used mainly to avoid overloading your site with a lot of crawl requests and save server bandwidth.

In this way, you can easily block unnecessary pages for crawling by allowing the important pages and saving server bandwidth.

The robots.txt file is a part of the robots exclusion protocol (REP), a group of web standards that regulate how robots or web crawlers crawl the web, access and index content, and serve that content up to users.

Usually, the robots.txt file is added in the root directory of the website and can easily access with a URL Like this.

https://example.com/robots.txt

So, you can easily check the robots.txt file of your Blogger website by adding the robots.txt after the homepage URL as shown in the above example.

Structure of default Robots.txt file

The basic format of a robots.txt file is like this

User-agent: [user-agent name]Disallow: [URL string not to be crawled]

Here one robots file can contain multiple lines of user agents and directives (i.e., disallows, allows, crawl-delays, etc.).

Here each set of user-agent directives is written as a discrete set, separated by a line break.

There are five common terms used in the robots.txt file.

• User-agent: It specifies the web crawler to which you’re giving crawl instructions (usually a search engine).
• Disallow: The command used to tell a user-agent not to crawl a particular URL. Only one “Disallow:” line is allowed for each URL.
• Allow: (Only applicable for Googlebot): This command tells Googlebot to access a page or subfolder even though its parent page or subfolder is disallowed.
• Crawl-delay: It specifies how many seconds a web crawler should wait before loading and crawling another page content. It is used to reduce the burden on the hosting server.
• Sitemap: It is Used to instruct web crawlers to crawl XML sitemap(s) associated with this URL. Note this command is only supported by Google, Ask, Bing, and Yahoo.

How to enable robots.txt file in Blogger website?

To enable the robots.txt file in Blogger follow the below steps.

Step-1: Go to Blogger settings and search for Crawlers and indexing option.

Step-2: Here turn on the “Enable custom robots.txt” option.

Step-3: Now click on the below option and paste the below code in the custom robots.txt field.

User-agent: *
Disallow: /search
Disallow: /category/
Disallow: /tag/
Allow: /
Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500

Step-4: Now save the code and the robots.txt file is added to your Blogger website.

Now you can check if this is implemented properly or not by accessing the URL. (https://www.yourdomain.com/robots.txt)

If you have more than 500 pages in the sitemap then you can add multiple sitemap URLs in the robots.txt file. Just change the start-index value like 501, 1001, etc.

Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=501&max-results=500

Now you have set up the custom robots.txt file on your Blogger website, you can set up custom robots header tags.

Just enable this option and click on the Homepage tags and select all and noodp and save the settings.

Home page tags	all, noodp
Archive and search page tags	noindex, noodp
Post and page tags	all, noodp

Why do you need robots.txt?

• Preventing duplicate content from appearing in SERPs
• It helps in blocking private sites like stagging sites
• Specifying the location of sitemap(s)
• You can preventing search engines from indexing certain files on your website (Like Premium images or PDFs, etc.)
• Specifying a crawl delay in order to prevent your servers from being overloaded when crawlers load multiple pieces of content at once

I hope this article helps you enable the robots.txt file on your blogger website and solve indexing-related problems.

If you want to learn more about Blogging & SEO, you can follow our YouTube channel.