Spock's Web Crawler

As part of Spock's mission to index every single human being on the planet, we have developed a crawler to collect pages all over the Internet. Spock's crawler has been designed and tested with the following politeness guidelines:

• Strict observance of the robots.txt standard. (Use 'spock' to refer to it specifically.)
• Crawling of low-traffic sites at a slower rate than high-traffic sites.

If you find that our crawler is disobeying your robots.txt, crawling at a excessive rate, or causing any other problems, please contact us at crawl@corp.spock.com, providing as many details as possible.