Robots.txt Generator
Generate a clean robots.txt file from simple inputs — user-agent groups with Allow/Disallow rules, crawl-delay, sitemap URLs, and one-click presets to allow all, block all, or block AI crawlers.
The Robots.txt Generator builds a valid robots.txt file for your website from simple inputs. Add one or more user-agent groups with Allow and Disallow paths, set an optional crawl-delay, list your sitemap URLs, or start from a preset — allow all robots, block all robots, or block AI crawlers — then copy or download a clean robots.txt to place at the root of your domain. Everything runs in your browser.
What is a robots.txt file?
robots.txt is a plain-text file that lives at the root of your domain (https://example.com/robots.txt) and tells search-engine crawlers which parts of your site they may or may not request. It follows the Robots Exclusion Protocol: each block starts with a User-agent line naming the crawler, followed by Disallow and Allow rules that match URL paths. robots.txt controls crawling, not indexing — to keep a page out of search results, use a noindex meta tag or an HTTP header instead.
How to create a robots.txt file
Fill in the user-agent groups and sitemap URLs, or pick a preset, then copy the result into a file named robots.txt at your site root.
- Pick a preset (allow all, block all, block AI crawlers) or leave it on Custom.
- For each user-agent group, enter the crawler name (use * for all robots) and the paths to Disallow or Allow, one per line.
- Optionally add a Crawl-delay in seconds and list your sitemap URLs.
- Click Copy or Download, then upload the file as robots.txt to the root of your domain so it is reachable at https://yourdomain.com/robots.txt.
Allow all vs. block all
To let every crawler access your whole site, use a single group with 'User-agent: *' and an empty Disallow line — this is the most permissive robots.txt. To block all crawling, use 'User-agent: *' with 'Disallow: /', which asks compliant robots to skip every path. Remember that disallowing a URL only prevents well-behaved crawlers from requesting it; it does not guarantee the page stays out of search results or hides it from people who have the link.
Blocking AI crawlers and specific bots
You can target individual crawlers by name in their own User-agent group. The 'Block AI crawlers' preset adds Disallow rules for common AI training and answer bots such as GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, and Bytespider, while still allowing normal search engines through a final 'User-agent: *' group. You can also add a Crawl-delay to ask aggressive bots to slow down between requests, though not every crawler honors it.
Why use this robots.txt generator?
- One-click presets for allow-all, block-all, and block-AI-crawlers, plus full custom control.
- Add unlimited user-agent groups, each with its own Allow, Disallow, and Crawl-delay rules.
- List multiple Sitemap directives — one per line — appended automatically to the output.
- Live preview with Copy and Download buttons so you can save a ready-to-upload robots.txt file.
- Runs entirely in your browser — nothing is uploaded, and the file is generated locally on your device.
Frequently asked questions
Where do I put the robots.txt file?
It must live at the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers only look at that exact location — a robots.txt in a subfolder is ignored. Each subdomain (for example blog.yourdomain.com) needs its own robots.txt.
Does Disallow remove a page from Google?
No. robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it, just without a description. To keep a page out of the index, allow it to be crawled and add a noindex meta tag or X-Robots-Tag header, or protect it behind authentication.
How do I block a specific bot but allow others?
Add a dedicated user-agent group with that bot's name and a 'Disallow: /' rule, then keep a separate 'User-agent: *' group with an empty Disallow for everyone else. The most specific matching group wins for each crawler.
Should I add my sitemap to robots.txt?
Yes, it is good practice. A Sitemap directive points crawlers to your XML sitemap and is independent of user-agent groups. You can list several Sitemap lines — just enter one full URL per line in the Sitemaps box.
Is this robots.txt generator free and private?
Yes. It is completely free and runs entirely in your browser with JavaScript. Nothing you type is uploaded or stored — the robots.txt is generated locally and is yours to copy or download.