# Note: Robots that access TWiki.org at a high rate are # automatically added to a blacklist and will not be able # to access content anymore. Download the TWiki package, # it contains all TWiki.org pages you need to get started # with your own TWiki. User-agent: iecrawler User-agent: ie_crawler User-agent: e-SocietyRobot Disallow: / User-agent: msnbot Crawl-delay: 120 Disallow: /twiki/bin/attach Disallow: /twiki/bin/changes Disallow: /twiki/bin/edit Disallow: /twiki/bin/geturl Disallow: /twiki/bin/installpasswd Disallow: /twiki/bin/login Disallow: /twiki/bin/logon Disallow: /twiki/bin/mailnotify Disallow: /twiki/bin/manage Disallow: /twiki/bin/oops Disallow: /twiki/bin/passwd Disallow: /twiki/bin/preview Disallow: /twiki/bin/rdiff Disallow: /twiki/bin/register Disallow: /twiki/bin/rename Disallow: /twiki/bin/save Disallow: /twiki/bin/search Disallow: /twiki/bin/statistics Disallow: /twiki/bin/upload Disallow: /twiki/bin/viewauth Disallow: /twiki/bin/viewfile Disallow: /twiki/bin/view/Sandbox Disallow: /twiki/bin/view/*? Disallow: /twiki/bin/survey Disallow: /twiki/bin/twikicvs Disallow: /twiki/bin/awstats/ Disallow: /twiki/bin/b/bin/ Disallow: /twiki/bin/view/Main/SearchResults* Disallow: /twiki/bin/view/Example/SpringMetaForm Disallow: /twiki/bin/view/Example/OpenSourceLicenseList Disallow: /twiki/bin/view/Example/SpringCoreVersionList Disallow: /twiki/bin/view/Example/SpringCoreList Disallow: /twiki/bin/view/Example/SpringProjectList Disallow: /twiki/bin/view/Example/OpenSourceProjectList Disallow: /twiki/bin/view/Example/*Comments Disallow: /simple-form/ Disallow: /simple-security/ Disallow: /simple-webflow/ Disallow: /swf-subflow/ Disallow: /simple-gwt/ User-agent: Slurp # Custom extension: (see http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html) Crawl-delay: 120 Disallow: /twiki/bin/attach Disallow: /twiki/bin/changes Disallow: /twiki/bin/edit Disallow: /twiki/bin/geturl Disallow: /twiki/bin/installpasswd Disallow: /twiki/bin/login Disallow: /twiki/bin/logon Disallow: /twiki/bin/mailnotify Disallow: /twiki/bin/manage Disallow: /twiki/bin/oops Disallow: /twiki/bin/passwd Disallow: /twiki/bin/preview Disallow: /twiki/bin/rdiff Disallow: /twiki/bin/register Disallow: /twiki/bin/rename Disallow: /twiki/bin/save Disallow: /twiki/bin/search Disallow: /twiki/bin/statistics Disallow: /twiki/bin/upload Disallow: /twiki/bin/viewauth Disallow: /twiki/bin/viewfile Disallow: /twiki/bin/view/Sandbox Disallow: /twiki/bin/view/*? Disallow: /twiki/bin/survey Disallow: /twiki/bin/twikicvs Disallow: /twiki/bin/awstats/ Disallow: /twiki/bin/b/bin/ Disallow: /twiki/bin/view/Main/SearchResults* Disallow: /twiki/bin/view/Example/SpringMetaForm Disallow: /twiki/bin/view/Example/OpenSourceLicenseList Disallow: /twiki/bin/view/Example/SpringCoreVersionList Disallow: /twiki/bin/view/Example/SpringCoreList Disallow: /twiki/bin/view/Example/SpringProjectList Disallow: /twiki/bin/view/Example/OpenSourceProjectList Disallow: /twiki/bin/view/Example/*Comments Disallow: /simple-form/ Disallow: /simple-security/ Disallow: /simple-webflow/ Disallow: /swf-subflow/ Disallow: /simple-gwt/ User-agent: * Crawl-delay: 120 Disallow: /twiki/bin/attach Disallow: /twiki/bin/changes Disallow: /twiki/bin/edit Disallow: /twiki/bin/geturl Disallow: /twiki/bin/installpasswd Disallow: /twiki/bin/login Disallow: /twiki/bin/logon Disallow: /twiki/bin/mailnotify Disallow: /twiki/bin/manage Disallow: /twiki/bin/oops Disallow: /twiki/bin/passwd Disallow: /twiki/bin/preview Disallow: /twiki/bin/rdiff Disallow: /twiki/bin/register Disallow: /twiki/bin/rename Disallow: /twiki/bin/save Disallow: /twiki/bin/search Disallow: /twiki/bin/statistics Disallow: /twiki/bin/upload Disallow: /twiki/bin/viewauth Disallow: /twiki/bin/viewfile Disallow: /twiki/bin/view/Sandbox Disallow: /twiki/bin/view/*? Disallow: /twiki/bin/survey Disallow: /twiki/bin/twikicvs Disallow: /twiki/bin/awstats/ Disallow: /twiki/bin/b/bin/ Disallow: /twiki/bin/view/Main/SearchResults* Disallow: /twiki/bin/view/Example/SpringMetaForm Disallow: /twiki/bin/view/Example/OpenSourceLicenseList Disallow: /twiki/bin/view/Example/SpringCoreVersionList Disallow: /twiki/bin/view/Example/SpringCoreList Disallow: /twiki/bin/view/Example/SpringProjectList Disallow: /twiki/bin/view/Example/OpenSourceProjectList Disallow: /twiki/bin/view/Example/*Comments Disallow: /twiki/bin/view/TWiki/* Disallow: /simple-form/ Disallow: /simple-security/ Disallow: /simple-webflow/ Disallow: /swf-subflow/ Disallow: /simple-gwt/