Robots.txt Generator
Generate robots.txt rules with crawler allow, disallow, crawl-delay, and sitemap entries.
Loading tool...
Create search crawler rules
Robots.txt Generator helps create crawler rules for the file that lives at your site root. Build Allow and Disallow paths, add a sitemap URL, and copy a clean robots.txt file for SEO and launch checks.
Useful For
- Block admin, API, or temporary paths from crawling
- Document public paths and sitemap location
- Generate blocking rules for staging websites
Good To Know
Robots.txt guides crawler behavior. To remove already indexed pages from search results, use noindex or search console tools as well.
Developer Check
Generated code, regexes, meta tags, and config snippets should be treated as drafts and tested again in your runtime, security policy, and deployment environment.
How It Works
Inputs are calculated or converted in the browser whenever possible. Compare important results with the original source before using them. Reviewed on Jul 2, 2026.
Robots.txt checks
Crawler rules
- Do not confuse full-site blocking with path-specific blocking
- Review Allow and Disallow priority by user-agent
- Disallow controls crawling and does not guarantee removal of already indexed URLs
Sitemap
- Use the canonical sitemap.xml absolute URL
- Make sure staging or development URLs are not included
- When using multiple sitemaps, keep sitemap index and robots.txt on the same canonical domain
Before deploy
- Validate with a search console robots tester
- Check sample important URLs are not accidentally blocked
- Review admin, search result, and filter URLs separately from revenue pages
After deploy
- Confirm https://domain.com/robots.txt returns 200
- Compare with sitemap URLs submitted to Google, Bing, and Naver
- Use noindex or authentication too for pages that must not be indexed
Yes. Draft robots.txt rules with user-agent, allow, disallow, and sitemap URL entries.
It helps control crawling but does not guarantee deindexing. Use noindex, authentication, and server rules for sensitive pages.
Place it at the domain root, usually /robots.txt, and verify the URL after deployment.