# Alle Bots User-agent: Googlebot Disallow: User-agent: Googlebot-Image Disallow: User-agent: msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) Disallow: User-agent: MSNBot Disallow: User-agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm) Disallow: User-agent: seekbot Disallow: User-agent: Slurp Disallow: User-agent: Technoratibot/0.7 Disallow: User-agent: WordPress/2.1.2 Disallow: User-agent: Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/2.2 Disallow: User-agent: Yahoo! Slurp Disallow: User-agent: AskJeeves Disallow: # Verbotene Bots User-agent: ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: TMCrawler Disallow: / User-agent: aconon Index Disallow: / User-agent: ActiveAgent Disallow: / User-agent: Alexibot Disallow: / User-agent: A-Online Search Disallow: / User-agent: Aqua_Products Disallow: / User-agent: Arachnoidea Disallow: / User-agent: Arachnophilia Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: asterias Disallow: / User-agent: ATN_Worldwide Disallow: / User-agent: autoemailspider Disallow: / User-agent: b2w 0.1 Disallow: / User-agent: b2w Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot 1.0 Disallow: / User-agent: BackDoorBot Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: BackWeb Disallow: / User-agent: baiduspider Disallow: / User-agent: BecomeBot Disallow: / User-agent: Black Hole Disallow: / User-agent: BlackWidow Disallow: / User-agent: BlowFish 1.0 Disallow: / User-agent: BlowFish Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: boitho.com-dc Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: BotALot Disallow: / User-agent: BotRightHere Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker /1.0 Disallow: / User-agent: CherryPicker 1.0 Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite 1.0 Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE 1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: ChinaClaw Disallow: / User-agent: Collector Disallow: / User-agent: CoolBot Disallow: / User-agent: Copernic Disallow: / User-agent: Copier Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: Crawler V 0.2.1 admin@crawler.de Disallow: / User-agent: Crescent Internet ToolPak HTTPOLE Control v.1.0 Disallow: / User-agent: Crescent Disallow: / User-agent: Deweb/1.01 Disallow: / User-agent: DISCo Pump 3.1 Disallow: / User-agent: DISCo Pump Disallow: / User-agent: DISCo Disallow: / User-agent: DittoSpyder Disallow: / User-agent: dloader(naverrobot)/1.0 Disallow: / User-agent: DOC Disallow: / User-agent: Download Demon Disallow: / User-agent: Download Ninja Disallow: / User-agent: Download Wonder Disallow: / User-agent: Downloader Disallow: / User-agent: Drip Disallow: / User-agent: dumbot Disallow: / User-agent: eCatch Disallow: / User-agent: EirGrabber Disallow: / User-agent: EmailCollector 1.0 Disallow: / User-agent: emailcollector Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf 1.00 Disallow: / User-agent: EmailWolf Disallow: / User-agent: Enterprise_Search Disallow: / User-agent: Enterprise_Search/1.0 Disallow: / User-agent: EroCrawler Disallow: / User-agent: es Disallow: / User-agent: Eule-Robot Disallow: / User-agent: Express WebPictures Disallow: / User-agent: ExtractorPro Disallow: / User-agent: EyeNetIE Disallow: / User-agent: FairAd Client Disallow: / User-agent: FAST Crawler (crawler at fast.no) Disallow: / User-agent: FAST Disallow: / User-agent: Fetch Disallow: / User-agent: fido/1.0 Disallow: / User-agent: FileHound Disallow: / User-agent: Firefly/1.0 Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: FlashGet Disallow: / User-agent: Foobot Disallow: / User-agent: FreeFind Disallow: / User-agent: Gaisbot Disallow: / User-agent: GetRight Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: GetSmart Disallow: / User-agent: GetWeb! Disallow: / User-agent: Gigabot Disallow: / User-agent: Go!Zilla Disallow: / User-agent: Go-Ahead-Got-It Disallow: / User-agent: gotit Disallow: / User-agent: Grabber Disallow: / User-agent: grabber Disallow: / User-agent: GrabNet Disallow: / User-agent: Grafula Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: Gulliver/1.3 Disallow: / User-agent: Harvest 1.5 Disallow: / User-agent: Harvest Disallow: / User-agent: Harvest/1.4.pl2 Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: Hatena Antenna Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: hloader #url: #note: Disallow: / User-agent: HLoader #url: #note: Disallow: / User-agent: HMView #url: #note: Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler #url: #note: Disallow: / User-agent: httplib #url: #note: Disallow: / User-agent: HTTrack #url: #note: Disallow: / User-agent: humanlinks #url: #note: Disallow: / User-agent: ia_archiver #url: #note: way back machine Disallow: User-agent: ia_archiver/1.6 #url: #note: Disallow: User-agent: IBM_Planetwide #url: #note: Disallow: / User-agent: Image Stripper #url: #note: Disallow: / User-agent: Image Sucker #url: #note: Disallow: / User-agent: Indy Library #url: #note: Disallow: / User-agent: InfoNaviRobot #url: #note: Disallow: / User-agent: InfoSeek Sidewinder/0.9 #url: #note: Disallow: / User-agent: InterGET #url: #note: Disallow: / User-agent: Internet Ninja #url: #note: Disallow: / User-agent: Iria Disallow: / User-agent: IRLbot Disallow: / User-agent: Iron33 #url: #note: Disallow: / User-agent: Iron33/1.0.2 #url: #note: Disallow: / User-agent: Jeeves #url: #note: Disallow: / User-agent: JennyBot #url: #note: Disallow: / User-agent: Jetbot #url: #note: Disallow: / User-agent: Jetbot/1.0 #url: #note: Disallow: / User-agent: JetCar #url: #note: Disallow: / User-agent: JOC Web Spider #url: #note: Disallow: / User-agent: JOC #url: #note: Disallow: / User-agent: JustView #url: #note: Disallow: / User-agent: k2spider #url: #note: Disallow: / User-agent: Kenjin Spider #url: #note: Disallow: / User-agent: Keyword Density #url: #note: Disallow: / User-agent: Keyword Density/0.9 #url: #note: Disallow: / User-agent: KIT-Fireball/2.0 #url: #note: Disallow: / User-agent: larbin #url: #note: Disallow: / User-agent: larbin #url: #note: Disallow: / User-agent: LexiBot #url: #note: Disallow: / User-agent: lftp #url: #note: Disallow: / User-agent: libWeb #url: #note: Disallow: / User-agent: libWeb/clsHTTP #url: #note: Disallow: / User-agent: libwww #url: #note: Disallow: / User-agent: likse #url: #note: Disallow: / User-agent: LinkextractorPro #url: #note: Disallow: / User-agent: linko #url: #note: Disallow: / User-agent: LinkScan #url: #note: Disallow: / User-agent: LinkScan/8.1a Unix #url: #note: Disallow: / User-agent: LinkWalker #url: #note: Disallow: / User-agent: LNSpiderguy #url: #note: Disallow: / User-agent: looksmart #url: #note: Disallow: / User-agent: lwp-trivial #url: #note: Disallow: / User-agent: lwp-trivial/1.34 #url: #note: Disallow: / User-agent: Lycos_Spider_(T-Rex) #url: #note: Disallow: User-agent: Magnet #url: #note: Disallow: / User-agent: Mag-Net #url: #note: Disallow: / User-agent: Mass Downloader #url: #note: Disallow: / User-agent: Mata Hari #url: #note: Disallow: / User-agent: MegaSheep #url: #note: Disallow: / User-agent: Memo #url: #note: Disallow: / User-agent: Microsoft URL Control - 5.01.4511 #url: #note: Disallow: User-agent: Microsoft URL Control - 6.00.8169 #url: #note: Disallow: User-agent: Microsoft URL Control #url: #note: Disallow: User-agent: Microsoft.URL.Control #url: #note: Disallow: User-agent: MIDown tool #url: #note: Disallow: / User-agent: MIIxpc #url: #note: Disallow: / User-agent: MIIxpc/4.2 #url: #note: Disallow: / User-agent: Mirror #url: #note: Disallow: / User-agent: Mister PiX #url: #note: Disallow: / User-agent: moget #url: #note: Disallow: / User-agent: moget/2.1 #url: #note: Disallow: / User-agent: mozilla #url: #note: Disallow: / User-agent: Mozilla #url: #note: Disallow: / User-agent: mozilla/3 #url: #note: Disallow: / User-agent: mozilla/4 #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) #url: #note: Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; AIRF) #url: #note: Disallow: / User-agent: mozilla/5 #url: #note: Disallow: / User-agent: MSIECrawler #url: #note: Disallow: / User-agent: my-heritrix-crawler(+http://mywebsite.com) #url: http://mywebsite.com #note: Disallow: / User-agent: naver #url: #note: Disallow: / User-agent: Navroad #url: #note: Disallow: / User-agent: NearSite #url: #note: Disallow: / User-agent: Net Vampire #url: #note: Disallow: / User-agent: NetAnts #url: #note: Disallow: / User-agent: NetMechanic #url: #note: Disallow: / User-agent: NetSpider #url: #note: Disallow: / User-agent: NetZIP #url: #note: Disallow: / User-agent: NICErsPRO #url: #note: Disallow: / User-agent: Ninja #url: #note: Disallow: / User-agent: NPBot #url: #note: Disallow: / User-agent: Nutch #url: #note: Disallow: / User-agent: Octopus #url: #note: Disallow: / User-agent: Offline Explorer #url: #note: Disallow: / User-agent: Offline Navigator #url: #note: Disallow: / User-agent: OmniExplorer_Bot #url: #note: Disallow: / User-agent: Openbot #url: #note: Disallow: / User-agent: Openfind #url: #note: Disallow: / User-agent: Openfind data gathere #url: #note: Disallow: / User-agent: Openfind data gatherer #url: #note: Disallow: / User-agent: Oracle Ultra Search #url: #note: Disallow: / User-agent: PageGrabber #url: #note: Disallow: / User-agent: Papa Foto #url: #note: Disallow: / User-agent: pavuk #url: #note: Disallow: / User-agent: pcBrowser #url: #note: Disallow: / User-agent: PerMan #url: #note: Disallow: / User-agent: ProPowerBot #url: #note: Disallow: / User-agent: ProPowerBot/2.14 #url: #note: Disallow: / User-agent: ProWebWalker #url: #note: Disallow: / User-agent: psbot #url: #note: Disallow: / User-agent: Pump #url: #note: Disallow: / User-agent: Python-urllib #url: #note: Disallow: / User-agent: QueryN Metasearch #url: #note: Disallow: / User-agent: Radiation Retriever 1.1 #url: #note: Disallow: / User-agent: Radiation Retriever #url: #note: Disallow: / User-agent: Radiation #url: #note: Disallow: / User-agent: RealDownload #url: #note: Disallow: / User-agent: Reaper #url: #note: Disallow: / User-agent: Recorder #url: #note: Disallow: / User-agent: ReGet #url: #note: Disallow: / User-agent: RepoMonkey Bait Tackle/v1.01 #url: #note: Disallow: / User-agent: RepoMonkey #url: #note: Disallow: / User-agent: RMA #url: #note: Disallow: / User-agent: Robozilla #url: #note: Disallow: / User-agent: Roverbot #url: #note: Disallow: / User-agent: scooter #url: #note: Disallow: / User-agent: Scooter #url: #note: Disallow: / User-agent: Scrubby #url: #note: Disallow: / User-agent: searchpreview #url: #note: Disallow: / User-agent: Siphon #url: #note: Disallow: / User-agent: sitecheck.internetseer.com #url: #note: Disallow: / User-agent: SiteSnagger #url: #note: Disallow: / User-agent: SmartDownload #url: #note: Disallow: / User-agent: Snake #url: #note: Disallow: / User-agent: sootle #url: #note: Disallow: / User-agent: SpaceBison #url: #note: Disallow: / User-agent: SpankBot #url: #note: Disallow: / User-agent: spanner #url: #note: Disallow: / User-agent: Stanford Comp Sci #url: #note: Disallow: / User-agent: Stanford #url: #note: Disallow: / User-agent: Sucker #url: #note: Disallow: / User-agent: SuperBot #url: #note: Disallow: / User-agent: SuperHTTP #url: #note: Disallow: / User-agent: Surfbot #url: #note: Disallow: / User-agent: suzuran #url: #note: Disallow: / User-agent: Szukacz #url: #note: Disallow: / User-agent: Szukacz/1.4 #url: #note: Disallow: / User-agent: tAkeOut #url: #note: Disallow: / User-agent: Tarantula #url: #note: Disallow: / User-agent: Teleport Pro #url: #note: Disallow: / User-agent: Teleport Pro/1.29.1590 #url: #note: Disallow: / User-agent: Teleport Pro/1.29.1616 #url: #note: Disallow: / User-agent: Teleport Pro/1.29.1632 #url: #note: Disallow: / User-agent: Teleport Pro/1.29.1718 #url: #note: Disallow: / User-agent: Teleport #url: #note: Disallow: / User-agent: TeleportPro #url: #note: Disallow: / User-agent: TeleportPro #url: #note: Disallow: / User-agent: Telesoft #url: #note: Disallow: / User-agent: Teoma #url: #note: Disallow: / User-agent: The Intraformant #url: #note: Disallow: / User-agent: TheNomad #url: #note: Disallow: / User-agent: thesubot #url: #note: Disallow: / User-agent: TightTwatBot #url: #note: Disallow: / User-agent: Titan #url: #note: Disallow: / User-agent: toCrawl/UrlDispatcher #url: #note: Disallow: / User-agent: True_Robot #url: #note: Disallow: / User-agent: True_Robot/1.0 #url: #note: Disallow: / User-agent: turingos #url: #note: Disallow: / User-agent: UbiCrawler #url: #note: Disallow: / User-agent: Ultraseek #url: #note: Disallow: / User-agent: URL Control #url: #note: Disallow: / User-agent: URL_Spider_Pro #url: #note: Disallow: / User-agent: URLy Warning #url: #note: Disallow: / User-agent: Vacuum #url: #note: Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 #url: #note: Disallow: / User-agent: VCI #url: #note: Disallow: / User-agent: VoidEYE #url: #note: Disallow: / User-agent: Web Image Collector #url: #note: Disallow: / User-agent: Web Sucker #url: #note: Disallow: / User-agent: WebAuto #url: #note: Disallow: / User-agent: WebBandit 2.1 #url: #note: Disallow: / User-agent: WebBandit 3.50 #url: #note: Disallow: / User-agent: webbandit 4.00.0 #url: #note: Disallow: / User-agent: WebBandit #url: #note: Disallow: / User-agent: WebBandit/3.50 #url: #note: Disallow: / User-agent: WebCapture 2.0 #url: #note: Disallow: / User-agent: WebCopier v.2.2 #url: #note: Disallow: / User-agent: WebCopier v3.2a #url: #note: Disallow: / User-agent: WebCopier #url: #note: Disallow: / User-agent: WebCrawler_1.1 internet@bredband.net #url: http://www.bredband.net/ #note: a swedish provider Disallow: User-agent: WebEMailExtrac. #url: #note: Disallow: / User-agent: WebEMailExtractor 1.0B #url: #note: Disallow: / User-agent: WebEnhancer #url: #note: Disallow: / User-agent: WebFetch #url: #note: Disallow: / User-agent: WebGo IS #url: #note: Disallow: / User-agent: WebLeacher #url: #note: Disallow: / User-agent: WebmasterWorld Extractor #url: #note: Disallow: / User-agent: WebmasterWorld #url: #note: Disallow: / User-agent: WebmasterWorldForumBot #url: #note: Disallow: / User-agent: WebMoose #url: #note: Disallow: / User-agent: WebReaper #url: #note: Disallow: / User-agent: WebSauger #url: #note: Disallow: / User-agent: Website eXtractor #url: #note: Disallow: / User-agent: Website Quester #url: #note: Disallow: / User-agent: Website #url: #note: Disallow: / User-agent: Webster Pro #url: #note: Disallow: / User-agent: Webster #url: #note: Disallow: / User-agent: WebStripper #url: #note: Disallow: / User-agent: WebVac #url: #note: Disallow: / User-agent: WebWhacker #url: #note: Disallow: / User-agent: WebZip #url: #note: Disallow: / User-agent: WebZip #url: #note: Disallow: / User-agent: WebZip/4.0 #url: #note: Disallow: / User-agent: WebZIP/4.21 #url: #note: Disallow: / User-agent: WebZIP/5.0 #url: #note: Disallow: / User-agent: wget #url: #note: Disallow: / User-agent: Wget #url: #note: Disallow: / User-agent: Wget/1.5.3 #url: #note: Disallow: / User-agent: Wget/1.6 #url: #note: Disallow: / User-agent: Whacker #url: #note: Disallow: / User-agent: whatUseek_winona #url: #note: Disallow: / User-agent: WhoWhere #url: #note: Disallow: / User-agent: Widow #url: #note: Disallow: / User-agent: WWW-Collector #url: #note: Disallow: / User-agent: WWW-Collector-E #url: #note: Disallow: / User-agent: WWWOFFLE #url: #note: Disallow: / User-agent: WX_mail Disallow: / User-agent: Xaldon Disallow: / User-agent: Xaldon/WebSpider Disallow: / User-agent: Xenu Disallow: / User-agent: Xenu\'s Link Sleuth 1.1c Disallow: / User-agent: Xenu\'s Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Xenu's Disallow: / User-agent: yahoo-blogs/v3.9 Disallow: User-agent: yahoo-mmcrawler Disallow: User-agent: Zao Disallow: / User-agent: Zealbot Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: Zeus Disallow: / User-agent: ZyBORG Disallow: /