The robots.txt file is the text file which tell the search engines which page on your website to crawl.

Example

                                        

It Allow all web crawlers access to all content

User-agent: *

Disallow:

Suggest An Answer

No suggestions avaliable!