# robots.txt for http://www.delphion.com User-agent: * Disallow: /cgi-bin/ Disallow: /fcgi-bin/ Disallow: /gifcache/ Disallow: /art/ Disallow: /images/ Disallow: /icons/ Disallow: /order Disallow: /vote Disallow: /tdbs/image Disallow: /patlist Disallow: /citelink Disallow: /nav/ Disallow: /derwent/p/ #Disallow: /details # Yahoo Slurp consumes too many DIVs 2008-08-07 User-agent: Slurp Disallow: /details*/ Disallow: /details/* Disallow: /cgi-bin/ Disallow: /fcgi-bin/ Disallow: /gifcache/ Disallow: /art/ Disallow: /images/ Disallow: /icons/ Disallow: /order Disallow: /vote Disallow: /tdbs/image Disallow: /patlist Disallow: /citelink Disallow: /nav/ Disallow: /derwent/p/ # Googlebot consumes too many DIVs now User-agent: Googlebot Disallow: /cgi-bin/ Disallow: /fcgi-bin/ Disallow: /gifcache/ Disallow: /art/ Disallow: /images/ Disallow: /icons/ Disallow: /order Disallow: /vote Disallow: /tdbs/image Disallow: /patlist Disallow: /citelink Disallow: /nav/ Disallow: /derwent/p/ Disallow: /details # Avoid recursive hti/trailer.hti errors in /home/* crawl User-agent: Scooter/3.2 Disallow: /home/ Disallow: /cgi-bin/ Disallow: /fcgi-bin/ Disallow: /gifcache/ Disallow: /art/ Disallow: /images/ Disallow: /icons/ Disallow: /order Disallow: /vote Disallow: /tdbs/image Disallow: /patlist Disallow: /citelink Disallow: /nav/ Disallow: /derwent/p/ # Verity spider - seems to not restrict self User-agent: vspider/3.6 Disallow: / Disallow: /cgi-bin/ Disallow: /fcgi-bin/ Disallow: /gifcache/ Disallow: /art/ Disallow: /images/ Disallow: /icons/ Disallow: /order Disallow: /vote Disallow: /tdbs/image Disallow: /patlist Disallow: /citelink Disallow: /nav/ Disallow: /derwent/p/ User-agent: Mapuccino/1.0 Allow: /cgi-bin/patsearch Allow: /cgi-bin/dofilter.pl Allow: /cgi-bin/db2www.cmd/v4/hitlist_sql.d2w/report Disallow: /cgi-bin/ Disallow: /any2html.gifcache/ Disallow: /images Disallow: /icons Disallow: /order Disallow: /vote Disallow: /ibm Disallow: /patquery Disallow: /advquery Disallow: /conditions Disallow: /mstatus Disallow: /whatsnew Disallow: /claims Disallow: /deliver Disallow: /refurl/ Disallow: /refurl Disallow: /galleryarch Disallow: /inhouse Disallow: /info Disallow: /info2 Disallow: /liclink Disallow: /searhhelp Disallow: /langhelp Disallow: /boolexample Disallow: /respage Disallow: /comment Disallow: /home Disallow: /boolquery Disallow: /faq Disallow: /chart Disallow: /help Disallow: /image_sample Disallow: /login Disallow: /promo/