Sicherheit mit robots.txt erhöhen
Hier geben wir Dir einige Beispiele, wie Du die Sicherheit mit robots erhöhen kannst und dadurch gleichzeitig Ressourcen für tatsächliche Besucher deiner Webseite freigibst. Schliesse Datensammler und Internet-Kopierer von deiner Webseite aus. Nicht jede Suchmaschine oder Bot hält sich jedoch an die Anweisungen in deiner robots.txt Datei. Die htaccess Einstellungen können von den Datensammlern nicht umgangen werden! Es ist jedoch besser einige Einträge in der robots.txt zu setzen, als keinerlei Sicherheitsmaßnahmen dort zu ergreifen.
User-agent: ia_archiver User-agent: tcpiputilsbot User-agent: OutfoxBot/0.5 User-agent: complex_network_group User-agent: imparser09.yandex.ru User-agent: spider14.yandex.ru User-agent: yandex.ru User-agent: yandex User-agent: Alexibot User-agent: Baiduspider User-agent: Baiduspider-image User-agent: Baiduspider-video User-agent: Aqua_Products User-agent: BackDoorBot User-agent: BackDoorBot/1.0 User-agent: BPImageWalker/2.0 User-agent: Black.Hole User-agent: BlackWidow User-agent: BlowFish User-agent: BlowFish/1.0 User-agent: Bookmark search tool User-agent: Bot mailto:craftbot [at] yahoo [dot] com User-agent: BotALot User-agent: BotRightHere User-agent: BuiltBotTough User-agent: Bullseye User-agent: Bullseye/1.0 User-agent: BunnySlippers User-agent: Cegbfeieh User-agent: CheeseBot User-agent: CherryPicker User-agent: CherryPickerElite/1.0 User-agent: CherryPickerSE/1.0 User-agent: ChinaClaw User-agent: Copernic User-agent: CopyRightCheck User-agent: Crescent User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: Custo User-agent: DISCo User-agent: DISCo Pump 3.0 User-agent: DISCo Pump 3.2 User-agent: DISCoFinder User-agent: DittoSpyder User-agent: Download Demon User-agent: Download Demon/3.2.0.8 User-agent: Download Demon/3.5.0.11 User-agent: EirGrabber User-agent: EmailCollector User-agent: EmailSiphon User-agent: EmailWolf User-agent: EroCrawler User-agent: Express WebPictures User-agent: Express WebPictures (www.express-soft.com) User-agent: ExtractorPro User-agent: EyeNetIE User-agent: FairAd Client User-agent: Flaming AttackBot User-agent: FlashGet User-agent: FlashGet WebWasher 3.2 User-agent: Foobot User-agent: FrontPage User-agent: FrontPage [NC,OR] User-agent: Gaisbot User-agent: GetRight User-agent: GetRight/2.11 User-agent: GetRight/3.1 User-agent: GetRight/3.2 User-agent: GetRight/3.3 User-agent: GetRight/3.3.3 User-agent: GetRight/3.3.4 User-agent: GetRight/4.0.0 User-agent: GetRight/4.1.0 User-agent: GetRight/4.1.1 User-agent: GetRight/4.1.2 User-agent: GetRight/4.2 User-agent: GetRight/4.2b (Portuguxeas) User-agent: GetRight/4.2c User-agent: GetRight/4.3 User-agent: GetRight/4.5 User-agent: GetRight/4.5a User-agent: GetRight/4.5b User-agent: GetRight/4.5b1 User-agent: GetRight/4.5b2 User-agent: GetRight/4.5b3 User-agent: GetRight/4.5b6 User-agent: GetRight/4.5b7 User-agent: GetRight/4.5c User-agent: GetRight/4.5d User-agent: GetRight/4.5e User-agent: GetRight/5.0beta1 User-agent: GetRight/5.0beta2 User-agent: GetWeb! User-agent: Go!Zilla User-agent: Go!Zilla (www.gozilla.com) User-agent: Go!Zilla 3.3 (www.gozilla.com) User-agent: Go!Zilla 3.5 (www.gozilla.com) User-agent: Go-Ahead-Got-It User-agent: GrabNet User-agent: Grafula User-agent: HMView User-agent: HTTrack User-agent: HTTrack 3.0 User-agent: HTTrack 3.0x User-agent: HTTrack [NC,OR] User-agent: Harvest User-agent: Harvest/1.5 User-agent: Image Stripper User-agent: Image Sucker User-agent: Indy Library User-agent: Indy Library [NC,OR] User-agent: InfoNaviRobot User-agent: InterGET User-agent: Internet Ninja User-agent: Internet Ninja 4.0 User-agent: Internet Ninja 5.0 User-agent: Internet Ninja 6.0 User-agent: Iron33/1.0.2 User-agent: JOC Web Spider User-agent: JennyBot User-agent: JetCar User-agent: Kenjin Spider User-agent: Kenjin.Spider User-agent: Keyword Density/0.9 User-agent: Keyword.Density User-agent: LNSpiderguy User-agent: LeechFTP User-agent: LexiBot User-agent: LinkScan/8.1a Unix User-agent: LinkScan/8.1a.Unix User-agent: LinkWalker User-agent: LinkextractorPro User-agent: MIDown tool User-agent: MIIxpc User-agent: MIIxpc/4.2 User-agent: MSIECrawler User-agent: Mass Downloader User-agent: Mass Downloader/2.2 User-agent: Mata Hari User-agent: Mata.Hari User-agent: Microsoft URL Control User-agent: Microsoft URL Control - 5.01.4511 User-agent: Microsoft URL Control - 6.00.8169 User-agent: Microsoft.URL User-agent: Mister PiX User-agent: Mister PiX version.dll User-agent: Mister Pix II 2.01 User-agent: Mister Pix II 2.02a User-agent: Mister.PiX User-agent: NICErsPRO User-agent: NPBot User-agent: NPbot User-agent: Navroad User-agent: NearSite User-agent: Net Vampire User-agent: Net Vampire/3.0 User-agent: NetAnts User-agent: NetAnts/1.10 User-agent: NetAnts/1.23 User-agent: NetAnts/1.24 User-agent: NetAnts/1.25 User-agent: NetMechanic User-agent: NetSpider User-agent: NetZIP User-agent: NetZip Downloader 1.0 Win32(Nov 12 1998) User-agent: NetZip-Downloader/1.0.62 (Win32; Dec 7 1998) User-agent: NetZippy+(http:/www.innerprise.net/usp-spider.asp) User-agent: Octopus User-agent: Offline Explorer User-agent: Offline Explorer/1.2 User-agent: Offline Explorer/1.4 User-agent: Offline Explorer/1.6 User-agent: Offline Explorer/1.7 User-agent: Offline Explorer/1.9 User-agent: Offline Explorer/2.0 User-agent: Offline Explorer/2.1 User-agent: Offline Explorer/2.3 User-agent: Offline Explorer/2.4 User-agent: Offline Explorer/2.5 User-agent: Offline Navigator User-agent: Offline.Explorer User-agent: Openbot User-agent: Openfind User-agent: Openfind data gatherer User-agent: Oracle Ultra Search User-agent: PageGrabber User-agent: Papa Foto User-agent: PerMan User-agent: ProPowerBot/2.14 User-agent: ProWebWalker User-agent: Python-urllib User-agent: QueryN Metasearch User-agent: QueryN.Metasearch User-agent: RMA User-agent: Radiation Retriever 1.1 User-agent: ReGet User-agent: RealDownload User-agent: RealDownload/4.0.0.40 User-agent: RealDownload/4.0.0.41 User-agent: RealDownload/4.0.0.42 User-agent: RepoMonkey User-agent: RepoMonkey Bait & Tackle/v1.01 User-agent: SiteSnagger User-agent: SlySearch User-agent: SmartDownload User-agent: SmartDownload/1.2.76 (Win32; Apr 1 1999) User-agent: SmartDownload/1.2.77 (Win32; Aug 17 1999) User-agent: SmartDownload/1.2.77 (Win32; Feb 1 2000) User-agent: SmartDownload/1.2.77 (Win32; Jun 19 2001) User-agent: SpankBot User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux User-agent: SuperBot User-agent: SuperBot/3.0 (Win32) User-agent: SuperBot/3.1 (Win32) User-agent: SuperHTTP User-agent: SuperHTTP/1.0 User-agent: Surfbot User-agent: Szukacz/1.4 User-agent: Teleport User-agent: Teleport Pro User-agent: Teleport Pro/1.29 User-agent: Teleport Pro/1.29.1590 User-agent: Teleport Pro/1.29.1634 User-agent: Teleport Pro/1.29.1718 User-agent: Teleport Pro/1.29.1820 User-agent: Teleport Pro/1.29.1847 User-agent: TeleportPro User-agent: Telesoft User-agent: The Intraformant User-agent: The.Intraformant User-agent: TheNomad User-agent: TightTwatBot User-agent: Titan User-agent: True_Robot User-agent: True_Robot/1.0 User-agent: TurnitinBot User-agent: TurnitinBot/1.5 User-agent: URL Control User-agent: URL_Spider_Pro User-agent: URLy Warning User-agent: URLy.Warning User-agent: VCI User-agent: VCI WebViewer VCI WebViewer Win32 User-agent: VoidEYE User-agent: WWW-Collector-E User-agent: WWWOFFLE User-agent: Web Image Collector User-agent: Web Sucker User-agent: Web.Image.Collector User-agent: WebAuto User-agent: WebAuto/3.40 (Win98; I) User-agent: WebBandit User-agent: WebBandit/3.50 User-agent: WebCapture 2.0 User-agent: WebCopier User-agent: WebCopier v.2.2 User-agent: WebCopier v2.5 User-agent: WebCopier v2.6 User-agent: WebCopier v2.7a User-agent: WebCopier v2.8 User-agent: WebCopier v3.0 User-agent: WebCopier v3.0.1 User-agent: WebCopier v3.2 User-agent: WebCopier v3.2a User-agent: WebEMailExtrac.* User-agent: WebEnhancer User-agent: WebFetch User-agent: WebGo IS User-agent: WebLeacher User-agent: WebReaper User-agent: WebReaper [info [at] webreaper [dot] net] User-agent: WebReaper [webreaper [at] otway [dot] com] User-agent: WebReaper v9.1 - www.otway.com/webreaper User-agent: WebReaper v9.7 - www.webreaper.net User-agent: WebReaper v9.8 - www.webreaper.net User-agent: WebReaper vWebReaper v7.3 - www,otway.com/webreaper User-agent: WebSauger User-agent: WebSauger 1.20b User-agent: WebSauger 1.20j User-agent: WebSauger 1.20k User-agent: WebStripper User-agent: WebStripper/2.03 User-agent: WebStripper/2.10 User-agent: WebStripper/2.12 User-agent: WebStripper/2.13 User-agent: WebStripper/2.15 User-agent: WebStripper/2.16 User-agent: WebStripper/2.19 User-agent: WebWhacker User-agent: WebZIP User-agent: WebZIP/2.75 (http:/www.spidersoft.com) User-agent: WebZIP/3.65 (http:/www.spidersoft.com) User-agent: WebZIP/3.80 (http:/www.spidersoft.com) User-agent: WebZIP/4.0 (http:/www.spidersoft.com) User-agent: WebZIP/4.1 (http:/www.spidersoft.com) User-agent: WebZIP/4.21 User-agent: WebZIP/4.21 (http:/www.spidersoft.com) User-agent: WebZIP/5.0 User-agent: WebZIP/5.0 (http:/www.spidersoft.com) User-agent: WebZIP/5.0 PR1 (http:/www.spidersoft.com) User-agent: WebZip User-agent: WebZip/4.0 User-agent: WebmasterWorldForumBot User-agent: Website Quester User-agent: Website Quester - www.asona.org User-agent: Website Quester - www.esalesbiz.com/extra/ User-agent: Website eXtractor User-agent: Website eXtractor (http:/www.asona.org) User-agent: Website.Quester User-agent: Webster Pro User-agent: Wget User-agent: Wget/1.10.2 User-agent: Wget/1.5.2 User-agent: Wget/1.5.3 User-agent: Wget/1.6 User-agent: Wget/1.7 User-agent: Wget/1.8 User-agent: Wget/1.8.1 User-agent: Wget/1.8.1+cvs User-agent: Wget/1.8.2 User-agent: Wget/1.9-beta User-agent: Widow User-agent: Xaldon WebSpider User-agent: Xaldon WebSpider 2.5.b3 User-agent: Xenu's User-agent: Xenu's Link Sleuth 1.1c User-agent: Zeus User-agent: Zeus 11389 Webster Pro V2.9 Win32 User-agent: Zeus 11652 Webster Pro V2.9 Win32 User-agent: Zeus 18018 Webster Pro V2.9 Win32 User-agent: Zeus 26378 Webster Pro V2.9 Win32 User-agent: Zeus 30747 Webster Pro V2.9 Win32 User-agent: Zeus 32297 Webster Pro V2.9 Win32 User-agent: Zeus 39206 Webster Pro V2.9 Win32 User-agent: Zeus 41641 Webster Pro V2.9 Win32 User-agent: Zeus 44238 Webster Pro V2.9 Win32 User-agent: Zeus 51070 Webster Pro V2.9 Win32 User-agent: Zeus 51674 Webster Pro V2.9 Win32 User-agent: Zeus 51837 Webster Pro V2.9 Win32 User-agent: Zeus 63567 Webster Pro V2.9 Win32 User-agent: Zeus 6694 Webster Pro V2.9 Win32 User-agent: Zeus 71129 Webster Pro V2.9 Win32 User-agent: Zeus 82016 Webster Pro V2.9 Win32 User-agent: Zeus 82900 Webster Pro V2.9 Win32 User-agent: Zeus 84842 Webster Pro V2.9 Win32 User-agent: Zeus 90872 Webster Pro V2.9 Win32 User-agent: Zeus 94934 Webster Pro V2.9 Win32 User-agent: Zeus 95245 Webster Pro V2.9 Win32 User-agent: Zeus 95351 Webster Pro V2.9 Win32 User-agent: Zeus 97371 Webster Pro V2.9 Win32 User-agent: Zeus Link Scout User-agent: asterias User-agent: b2w/0.1 User-agent: cosmos User-agent: eCatch User-agent: eCatch/3.0 User-agent: hloader User-agent: httplib User-agent: humanlinks User-agent: larbin User-agent: larbin (samualt9 [at] bigfoot [dot] com) User-agent: larbin samualt9 [at] bigfoot [dot] com User-agent: larbin_2.6.2 (kabura [at] sushi [dot] com) User-agent: larbin_2.6.2 (larbin2 [dot] 6 [dot] 2 [at] unspecified [dot] mail) User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu) User-agent: larbin_2.6.2 (vitalbox1 [at] hotmail [dot] com) User-agent: larbin_2.6.2 kabura [at] sushi [dot] com User-agent: larbin_2.6.2 larbin2 [dot] 6 [dot] 2 [at] unspecified [dot] mail User-agent: larbin_2.6.2 larbin [at] correa [dot] org User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu User-agent: larbin_2.6.2 vitalbox1 [at] hotmail [dot] com User-agent: libWeb/clsHTTP User-agent: lwp-trivial User-agent: lwp-trivial/1.34 User-agent: moget User-agent: moget/2.1 User-agent: pavuk User-agent: pcBrowser User-agent: psbot User-agent: searchpreview User-agent: spanner User-agent: suzuran User-agent: tAkeOut User-agent: toCrawl/UrlDispatcher User-agent: turingos User-agent: webfetch/2.1.0 User-agent: wget Disallow: /
„User-agent: ia_archiver“ bewirkt das deine Webseite nicht mehr in der Wayback Maschine archiviert wird.
Sitemap: http://www.domain.de/page-sitemap.xml
Sitemap: http://www.domain.de/post-sitemap.xml
Sitemap: http://www.domain.de/category-sitemap.xml
User-agent: *
Disallow:
Disallow: /wp-
Allow: /wp-includes/
Allow: /wp-content/themes/MyTheme/
Allow: /wp-content/plugins/
Disallow: */trackback$
Disallow: */trackback/$
Disallow: /liesmich.html
Disallow: /xmlrpc.php
Disallow: *.pdf$
User-Agent: Googlebot
Allow: .js
Allow: .css
User-agent: grub-client
Disallow: /
User-agent: grub
Disallow: /
User-agent: looksmart
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: larbin
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: psbot
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: WebZip/4.0
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: RMA
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: asterias
Disallow: /
User-agent: httplib
Disallow: /
User-agent: turingos
Disallow: /
User-agent: spanner
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: WebBandit/3.50
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: Wget/1.6
Disallow: /
User-agent: Wget/1.5.3
Disallow: /
User-agent: Wget
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: moget
Disallow: /
User-agent: hloader
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: grub
Disallow: /
User-agent: looksmart
Disallow: /
User-agent: WebZip
Disallow: /
User-agent: larbin
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: psbot
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: URL_Spider_Pro
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: WebBandit
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: ProWebWalker
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Telesoft
Disallow: /
User-agent: Website Quester
Disallow: /
User-agent: moget/2.1
Disallow: /
User-agent: WebAuto
Disallow: /
User-agent: WebZip/4.0
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebSauger
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: TheNomad
Disallow: /
User-agent: WWW-Collector-E
Disallow: /
User-agent: RMA
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: asterias
Disallow: /
User-agent: httplib
Disallow: /
User-agent: turingos
Disallow: /
User-agent: spanner
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-agent: WebBandit/3.50
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: WebmasterWorldForumBot
Disallow: /
User-agent: SpankBot
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: URLy Warning
Disallow: /
User-agent: Wget/1.6
Disallow: /
User-agent: Wget/1.5.3
Disallow: /
User-agent: Wget
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: moget
Disallow: /
User-agent: hloader
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: True_Robot/1.0
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /
User-agent: VCI
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: Openfind data gathere
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Xenu�s Link Sleuth 1.1c
Disallow: /
User-agent: Xenu�s
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: Radiation Retriever 1.1
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: Web Image Collector
Disallow: /
User-agent: The Intraformant
Disallow: /
User-agent: True_Robot/1.0
Disallow: /
User-agent: True_Robot
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: ProPowerBot/2.14
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: toCrawl/UrlDispatcher
Disallow: /
User-agent: WebEnhancer
Disallow: /
User-agent: suzuran
Disallow: /
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow: /
User-agent: VCI
Disallow: /
User-agent: Szukacz/1.4
Disallow: /
User-agent: QueryN Metasearch
Disallow: /
User-agent: Openfind data gathere
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Xenu�s Link Sleuth 1.1c
Disallow: /
User-agent: Xenu�s
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /
User-agent: RepoMonkey
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Openbot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: searchpreview
Disallow: /
User-agent: Zeus Link Scout
Disallow: /
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /
User-agent: Webster Pro
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: Radiation Retriever 1.1
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Oracle Ultra Search
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: PerMan
Disallow: /
Beachte auch Änderungsanforderungen der jeweiligen Suchmaschine, wie z.B. die Richtlinien für Webmaster von Google. Welche Einstellungen verwendest Du?