# All robots will spider the domain User-agent: * Disallow: /i/ Crawl-delay: 10 User-agent: dotbot Disallow: /