User-agent: * Allow: / # Limit crawl rate to 10 requests per 60 seconds Request-rate: 10/60