Robots.txt Generator
Compose robots.txt rules and sitemap references.
Create robots.txt content with one or more User-agent sections, Allow and Disallow path lists, optional Crawl-delay, and Sitemap URL lines. Copy or download the result for your site root.
Limitations
robots.txt is a crawl hint, not access control — sensitive URLs still need authentication. Google largely ignores Crawl-delay; malicious bots may ignore the file entirely. Validate syntax and policy with your stack before production use.
Recommended tools
FAQ
No. It only asks well-behaved crawlers not to fetch URLs. Anyone can still open the page directly; use authentication or server rules for real protection.
Not always. If other sites link to the URL, Google may still index it without crawling, or show a URL-only result. Use noindex when you need to keep a page out of search results.
Google ignores Crawl-delay in robots.txt. Use Search Console rate settings or fix server performance instead. Some other bots may honor it, but support varies.