Angelsworld Web-Tool service 2009
 
  » Terug
 

Free Robots.txt Generator

Default -  All Robots are:  
Crawl-Delay:
     
Specific Search Robots: Google   googlebot
  MSN Search   msnbot
  Yahoo   yahoo-slurp
  Ask/Teoma   teoma
  GigaBlast   gigabot
  Scrub The Web   scrubby
  DMOZ Checker   robozilla
  Nutch   nutch
  Alexa/Wayback   ia_archiver
   
Specific Special Bots: Google Image   googlebot-image
  Yahoo MM   yahoo-mmcrawler
  MSN PicSearch   psbot
  SingingFish   asterias
  Yahoo Blogs   yahoo-blogs/v3.9
   
Restricted Directories: The path is relative to root and must contain a trailing "/"
 
 
 
 
 
   

Now, copy and paste this text into a blank text file called "robots.txt" (don't forget the "s" on the end of "robots") and put it in your root directory. Like all other files on your server, make sure it's permissions are set so that visitors (such as search engines) can read it.

© Design by Kiran - All Rights Reserved @ Angelsworld 2009