Robots.txt File benefits

Robots are crawler of search engines. A crawler is a program used by search engines to collect data from the internet. When a crawler visits a website, it picks over the entire website content or as allowed to and stores it in its data storage servers.

When we submit site to search engines, like google, bing etc., their Search Engine Bots when reaches to site they will follow all page links and crawl and index them, and then afterward website and its post, pages were shown in search engine results.

For WordPress site, robots.txt file has an important role for search engine bots, it helps how website pages will be crawled/searched as decided by robots.txt

As its name signifies robots.txt is a text file that would be in the website root directory. Search engine bots follow your website and crawl all pages that use robots.txt instructions as defined.

Crawling permissions will be defined in this file. User-agent is used for the type of search engine crawler.

A blog or website has not the only post as data, besides has pages, categories, tags, comments, etc. But all these things were not useful for the search engine. Generally, a blog has traffic came from search engine and has main Url (https://www.jaseir.com) with posts, pages, and images besides having archive, pagination, wp-admin like search engine is not important, here robots.txt, search engine bots gives instruction to access unnecessary pages to not crawl instructions.

Here if URL crawl permission is not granted by robots.txt then due to this reason URL is blocked for search engine.

Means which website with which web pages, Google or Bing will show or which not robots.txt file decides, to remove from search engine remove by blogger its the reason they fear creating this. 

If you have not created robots.txt file in the blog and to update, some basic rules need to understand for our blog we need perfect SEO optimized robots.txt file.

How to create WordPress Robots.txt

For answer WordPress blog default robots.txt file is used for our blog better performance and for SEO we need to customize robots.txt as per our opinion. 

WordPress default robots.txt

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Sitemap: [SITE URL]/sitemap.xml

Like as above robots.txt has code/syntax but most blogger use syntax without much understanding in the blog, it should need proper understanding before we use it with syntax: 

User-Agent: it is used for kind of Search Engines Crawlers/Bots we use to give instruction like goooglebot, bingbot

User-agent: * is used for search engine bots (Ex: googlebot, bingbot etc.) that crawls a website

User-agent: googlebot

here only Google bot has permission to crawl page

Allow: This tag permit to search engine bots for web pages and folder to crawl 

Disallow: This syntax not allow bots to crawl and index, thus can’t access by any bot

If you want that your site pages and directory to index then this below syntax is used by Blogger in robots.txt file 

User-agent: *

Disallow:

But below code will block all pages and directory to index

User-agent: *

Disallow: /

if you use Adsense then this code below need to use. This is for AdSense robots that manage the ads  

User-agent: Mediapartners-Google*

Allow: /

Example: If like below code in robots.txt file then what it signifies, lets see that:

User-agent: *

Allow: /wp-content/uploads/

Disallow: /wp-content/plugins/

Disallow: /wp- admin/

Disallow: /archives/

Disallow: /refer/

Sitemap: https://jaseir.com/sitemap.xml

Whatever files you use in WordPress for images to upload they were saved inside /wp-content/uploads/ directory and disallow not permits images and files to index WordPress plugin files, 

Disallow also disallow search bots to index WordPress admin area, category page, and affiliate links to crawl 

On adding a sitemap to Robots.txt file, search engine bots can easily crawl site pages easily

We can create in our website different kind of robots.txt file as per our need and its not compulsory that what I am using here in robots.txt code you will also use the same code

User-agent: *

Disallow: /cgi-bin/

Disallow: /wp-admin/

Disallow: /archives/

Disallow: /*?*

Disallow: /comments/feed/

Disallow: /refer/

Disallow: /index.php

Disallow: /wp-content/plugins/

User-agent: Mediapartners-Google*

Allow: /

User-agent: Googlebot-Image

Allow: /wp-content/uploads/

User-agent: Adsbot-Google

Allow: /

User-agent: Googlebot-Mobile

Allow: /

Sitemap: https://www.jaseir.com/sitemap.xml

Note: Here inside Sitemap you need to use your site links 

Disallow is used not to crawl pages

You can update robots.txt code with the help of Yoast SEO plugin or manually update the root directory of your WordPress site directory.

Yoast SEO > Tools > File Editor