What is Robot.txt file in SEO
Robot.txt is a text file that tells search engine robots which pages on your sites to crawl.
Its also tells web robots which pages not to crawl.
Steps to create Robot.txt file for blog
1.) Open website https://ctrlq.org/blogger/
2.) Put website in generate sitemap option and press enter
Steps to create Robot.txt file for blog
1.) Open website https://ctrlq.org/blogger/
2.) Put website in generate sitemap option and press enter
3.) After clicking enter your robot.txt file will generate |
| Add caption |
4.) you can copy all that blue portion and paste in blogger and click on save option
go in blogger setting option - Search Preference - Click on custom robot.txt Edit
and paste all that blue portion text there and click on save button.
Steps to Submit Updated robot.txt to Google
1.Put in google robots.txt tester and press enter
2. After press enter you will see interface like this
3. Click on 2 option which marked in blue after you will see this interface
1.Put in google robots.txt tester and press enter
2. After press enter you will see interface like this
3. Click on 2 option which marked in blue after you will see this interface
4.Paste in that box generated robot.txt data and click on submit.This submit function of robot.txt tester allow you to ask google to more quickly crawl and index a new robot.txt file for your site.
5.you can also test your website weather its block or not by putting your URL below
and click on Test.
Blocking all web crawlers from all content
User agent : *
Disallow : /
Allowing all web crawlers to access all content
User agent : *
Disallow :
