# Allow all robots to spider everything by disallowing nothing User-agent: * Crawl-Delay: 20 Disallow: