Robots.txt Generator

100% Free Seo Tools
ad

Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


ad

About Robots.txt Generator

Robots txt file is mainly to understand the search engine which pages to index and which pages not to index. The job of the spider is to crawl every web pages up the website and take necessary steps for the search engine.

In order to exclude the files to no-index in search engine, robots txt file plays a key role

Guidelines: Robots Txt Guidelines

Help Tool: Robots Txt Tester - This Tool will help you check robots file working properly or not.

Instructions to use Robots txt Generator file:

1) Hide directories: For this type of situation if you don't want to make visible directory lively, just specify the directory name. So that all the files in the directory are completely blocked by search engines.

Eg:

User-Agent: *
Disallow: /admin/

2) Disallow pages: If you want to hide or block particular pages, just specify the path of the page disallows. Once you generate the robots file, just copy the code and paste in the robots.txt file. Upload the robots.txt file in the top root directory of hosting.

Eg:

User-Agent: *
Disallow: /page1
Disallow: /page2

Where to upload robots txt file?

You need to upload robots txt file (robots.txt) in the top root directory of public HTML file. When search engines crawl the web pages it first find robots.txt files and take action accordingly to the robots direction to index pages in search engines.

Check out other tools: XML Sitemap Generator


ad
ad