The robots.txt file is the text file which tells the search engines which page on your website to crawl.

This is most popular SEO interview questions

BY Best Interview Question ON 29 Sep 2019

Example

It Allow all web crawlers access to all content

User-agent: *

Disallow: