Robots.txt Validator

What does a robots.txt validator check?

A robots.txt validator parses your file the way a crawler would and flags anything that breaks the Robots Exclusion Protocol or is likely to be ignored. This validator reports errors (lines that are malformed or in the wrong place) and warnings (lines that are technically parsable but probably not what you intended), so you can fix problems before search engines or AI crawlers read the file.

1. Unknown or misspelled directives, such as 'Disalow' instead of 'Disallow', with a suggested correction.
2. Disallow, Allow or Crawl-delay rules that appear before any User-agent line and therefore belong to no group.
3. Missing colons, empty User-agent names, and empty directive names.
4. Crawl-delay values that aren't a non-negative number.
5. Sitemap directives that are empty or use a relative path instead of a full https:// URL.

How to validate your robots.txt

Paste the file and validate it in three steps.

1. Copy the contents of your live robots.txt (usually at https://yourdomain.com/robots.txt) or the draft you're editing.
2. Paste it into the box and click Validate, or click Load sample to see how problems are reported.
3. Review the line-numbered errors and warnings, fix them in your source file, and re-validate until it reports no problems.

Errors vs. warnings

Errors are lines a compliant parser can't act on — a missing colon, an unknown directive, a rule that belongs to no User-agent group, or a Sitemap that isn't an absolute URL. These should always be fixed. Warnings are lines that parse but look suspicious, such as a Disallow path that doesn't start with '/' or a file with no User-agent directive at all. Warnings are worth reviewing even if crawlers tolerate them, because they often signal a typo or a rule that won't do what you expect.

Common robots.txt mistakes

The most frequent problems are ordering and URL issues. Crawlers read robots.txt as a series of groups, each beginning with one or more User-agent lines followed by that group's rules — putting a Disallow before the first User-agent leaves it attached to nothing. Sitemap directives are global and must be absolute URLs (https://example.com/sitemap.xml), not paths like /sitemap.xml. Remember that robots.txt controls crawling, not indexing: a Disallow line stops well-behaved crawlers from fetching a URL but does not guarantee the page stays out of search results.

Why use this robots.txt validator?

• Line-numbered errors and warnings so you can jump straight to each problem in your source file.
• Catches misspelled directives like 'Disalow' or 'Useragent' and suggests the correct spelling.
• Flags rules placed before any User-agent group, the single most common robots.txt mistake.
• Validates Sitemap directives as absolute URLs and checks Crawl-delay is a valid number.
• Runs entirely in your browser — your robots.txt is never uploaded or stored anywhere.

Frequently asked questions