# Allow Archiver
User-agent: ia_archiver
Allow: /


User-agent: Slurp
Crawl-delay: 60


User-agent: *


Sitemap: http://thebotnet.com/sitemap_index.xml.gz