### robots.txt for http://www.site.com/ ## ##User-agent: * ##Disallow: /cyberworld/map/ # This is an infinite virtual URL space ##Disallow: /tmp/ # these will soon disappear ## ## ##This example "/robots.txt" file specifies that no robots should visit any URL starting with ##"/cyberworld/map/", except the robot called "cybermapper": ## ## ### robots.txt for http://www.site.com/ ## ##User-agent: * ##Disallow: /cyberworld/map/ # This is an infinite virtual URL space ## ### Cybermapper knows where to go. ##User-agent: cybermapper ##Disallow: ## ##This example indicates that no robots should visit this site further: ## ### go away User-agent: * Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /htdig/ Disallow: /include/ Disallow: /Images/ Disallow: /logs/ Disallow: /logscribe/ Disallow: /access_test/ Disallow: /sbir_survey/ Disallow: /sbir/ Disallow: /SBIR00/ Disallow: /STTR00/ Disallow: /SBIR01/ Disallow: /SBIR02/ Disallow: /swishtestnew/ #13