Free Robots txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Free Robots txt Generator

Do I need a robots.txt file?

You may not even need to have a robots.txt file on your site. In fact, it is often the case you do not need one.

Reasons you may want to have a robots.txt file:

  1. You have content you want blocked from search engines
  2. You are developing a site that is live, but you do not want search engines to index it yet
  3. You want to fine tune access to your site from reputable robots
  4. You are using paid links or advertisements that need special instructions for robots
  5. They help you follow some Google guidelines in some situations

Reasons you may not want to have a robots.txt file:

  1. Your site is simple and error free and you want everything indexed
  2. You do not have any files you want or need to be blocked from search engines
  3. You do not find yourself in any of the situations listed in the above reasons to have a robots.txt file
  4. It is okay to not have a robots.txt file.

When you do not have a robots.txt file the search engine robots like Googlebot will have full access to your site. This is a normal and simple method that is very common.

Keys to robots.txt

  • If you use a robots.txt file, make sure it is being used properly
  • An incorrect robots.txt file can block Googlebot from indexing your page
  • Ensure you are not blocking pages or elements that Google needs to read, render and rank your pages