Robots.txt Validator — Check Syntax & Rules

Robots.txt Content

What is a Robots.txt?

A robots.txt validator checks whether your robots.txt file is correctly written according to the Robots Exclusion Protocol, the standard that governs how search engine crawlers interact with websites. A single syntax error in robots.txt can silently block an entire section of your website from being crawled, or fail to protect the pages you intended to hide from search engines.

The robots.txt file lives at the root of your domain, always at https://yoursite.com/robots.txt, and is one of the first files a search engine crawler reads when it visits your site. It is plain text, not XML or JSON, so there is no schema for parsers to validate against automatically. That means malformed directives are simply ignored by crawlers rather than throwing an error, making it easy to deploy a broken robots.txt without realising it for days or weeks.

This validator reads your robots.txt line by line, applying the same parsing rules that major crawlers use. It checks that every Disallow and Allow directive is preceded by a User-agent block, that path values begin with a forward slash, that Sitemap directives reference absolute URLs, and that no unrecognised directives are present. It also reports a summary, number of rule blocks, total Disallow and Allow rules, and Sitemap references, so you can confirm the file's shape matches your intentions.

For the corresponding sitemap file referenced by your robots.txt, our Sitemap Validator checks the XML structure and URL count. For individual URL format checks, the URL Validator is the right tool.

Why Use a Robots.txt Validator?

Robots.txt errors are silent by default. Google Search Console will surface crawl anomalies over time, but it does not parse your robots.txt and report structural errors in real time. A misconfigured file, one Disallow: / in the wrong block, a missing colon on a directive, or an Allow without a preceding User-agent, can go undetected until you notice unexpected drops in crawl coverage or missing pages in search results.

Common mistakes that this validator catches:

Disallow before User-agent, some generators or manual edits place a Disallow directive at the top of the file before any User-agent block. Most crawlers will ignore it entirely.
Misspelled directive, Useragent: instead of User-agent: is silently skipped by every crawler.
Relative Sitemap URL, Sitemap: /sitemap.xml looks reasonable but is invalid; Sitemap directives require an absolute URL including the protocol.
Non-slash Disallow paths, Disallow: private/ (missing leading slash) is not a valid path and is ignored by spec-compliant crawlers.

Validating after every deployment or CMS update is a low-effort habit that prevents a category of SEO bugs that are disproportionately hard to debug after the fact.

Who Should Use This Validator?

Web developers deploying a new site or migrating to a new CMS should validate the robots.txt immediately after go-live. Platform changes, moving from WordPress to a headless CMS, switching sitemap plugins, or changing URL structures, often regenerate the robots.txt in unexpected ways.

SEO professionals and consultants running a technical site audit routinely check robots.txt as one of the first steps. A misconfigured file is a high-severity finding that can explain a wide range of crawl and indexation issues.

Website owners who manage their own WordPress, Shopify, or other CMS site and have edited robots.txt directly or via a plugin. Plugins sometimes overwrite manual edits during updates, and it is worth re-validating after any plugin update.

DevOps and site reliability engineers who include robots.txt in deployment pipelines can use this validator to add a programmatic sanity check, paste the generated output and confirm it is valid before the deployment completes.

If your robots.txt references a sitemap, validate both files together: start here for the robots.txt, then use our Sitemap Validator for the sitemap XML.

What Does the Robots.txt Validator Check?

Valid / Invalid status, the primary result tells you immediately whether the file is syntactically correct. Any error makes the result invalid, with each error described specifically below the summary stats.

Rule blocks, the number of distinct User-agent groups in the file. Each block applies its Disallow and Allow rules to the crawlers listed in its User-agent lines. A file with zero rule blocks is invalid.

Disallow rules, the total count of Disallow directives across all blocks. A high count is not a problem in itself, but a count of zero on a file intended to restrict crawlers is worth checking.

Allow rules, the total count of Allow directives. These are typically used to carve out exceptions within a broader Disallow.

Sitemap directives, the number of Sitemap: lines present. Each must be an absolute http:// or https:// URL pointing to a sitemap file.

Errors, specific problems that make the file non-compliant: directives in the wrong order, invalid path values, bad Sitemap URLs, or empty User-agent values.

Warnings, non-fatal issues worth reviewing: rule blocks with no rules (a User-agent block containing neither Disallow nor Allow), and unknown directives that crawlers will silently ignore.

How to use this Robots.txt calculator

Get your robots.txt content, open https://yoursite.com/robots.txt in a browser, select all the text, and copy it. Alternatively, open the file directly from your server or CMS.
Paste it into the input field, the Robots.txt Content textarea accepts the raw text exactly as it appears in the file. There is no need to wrap it in quotes or modify the formatting.
Read the result, validation runs automatically as you type or paste. A green Valid badge means the file is syntactically correct; a red Invalid badge is followed by a list of specific errors with line numbers.
Fix any errors, each error message includes the line number and a description of what is wrong. Return to your file, fix the indicated line, and paste the updated version to re-validate.
Check the summary stats, even on a valid file, review the rule block count and Disallow count to confirm they match your expectations. An unexpectedly high Disallow count or a missing Sitemap directive can indicate a misconfiguration that is technically valid but not what you intended.
Validate your sitemap, if the file includes one or more Sitemap: directives, paste the sitemap XML into our Sitemap Validator to confirm the referenced file is also well-formed.

Robots.txt Format & Specification

A valid robots.txt file is plain text with one directive per line. Each directive is a field name, a colon, and a value:

# Comment lines start with #

User-agent: *
Disallow: /private/
Allow: /private/public-page/
Crawl-delay: 2

User-agent: Googlebot
Disallow: /staging/

Sitemap: https://example.com/sitemap.xml

Rules of the protocol:

Directive	Required?	Valid value
`User-agent`	Yes (must precede rules)	Crawler name or `*` for all
`Disallow`	At least one per block	Path starting with `/`, or empty (allow all)
`Allow`	No	Path starting with `/`
`Sitemap`	No	Absolute URL (http:// or https://)
`Crawl-delay`	No	Positive integer (seconds)

Path matching rules:

Disallow: /private/, blocks /private/ and all URLs that begin with /private/
Disallow: /, blocks the entire site
Disallow: (empty), allows the entire site; equivalent to no Disallow
* in a path matches any sequence of characters: Disallow: /*.pdf$ blocks all PDF URLs
$ at the end of a path anchors the match to the end of the URL

Block separation: an empty line ends a rule block. Directives after the empty line belong to the next block. Sitemap and Host directives are typically placed outside any rule block, at the end of the file.

Related Tools

You may also find these useful: XML Validator.

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file is a plain-text file placed at the root of a website (https://yoursite.com/robots.txt) that tells search engine crawlers which parts of the site they are and are not allowed to access. It follows the Robots Exclusion Protocol, a de facto standard supported by Google, Bing, and most other major crawlers. The file is publicly readable, anyone can view it, so it should never be used to protect sensitive content, only to manage crawler behaviour.

What does a robots.txt validator check?

This validator checks that your robots.txt file follows the Robots Exclusion Protocol correctly. It verifies that every Disallow and Allow directive is preceded by a User-agent directive, that path values start with a forward slash, that Sitemap URLs are absolute http(s) addresses, and that no unknown or misspelled directives are present. It also counts rule blocks, Disallow rules, Allow rules, and Sitemap entries so you can audit the file's coverage at a glance.

What is the difference between Disallow and Allow in robots.txt?

Disallow tells a crawler not to access the specified path or any URL that begins with that path. Allow is used to create exceptions within a Disallow rule, for example, you could Disallow an entire directory but Allow a specific file inside it. Most major crawlers, including Googlebot, honour both directives. When Allow and Disallow rules conflict for the same URL, the more specific (longer) path takes precedence; if both are the same length, the Allow wins.

Does Disallow: / block a page from being indexed?

Disallow: / tells crawlers not to crawl any page on your site, but it does not prevent pages from being indexed if they are linked to from other websites. Google can index a URL it has never crawled if it appears in an external link, showing a result without any snippet. To prevent indexing entirely, you need a noindex meta tag or HTTP header on the page itself, but that requires the page to be crawlable in the first place, which creates a catch-22 for blocked pages.

What is the wildcard (*) User-agent and when should I use it?

User-agent: * applies the following rules to all crawlers that are not covered by a more specific User-agent block. It acts as a catch-all. Most robots.txt files include at least one User-agent: * block to set a baseline policy for any crawler that visits the site. If you want to allow all crawlers unrestricted access, you still need the block, use User-agent: * followed by Disallow: (empty value) to explicitly permit everything.

What does an empty Disallow mean?

An empty Disallow: line (nothing after the colon) means the crawler is allowed to access the entire site. It is the conventional way to tell a search engine that it may crawl everything, equivalent to having no Disallow rules at all. Many CMS platforms generate this as the default robots.txt for public sites: User-agent: * on one line, then Disallow: on the next.

Can I have multiple User-agent lines in one block?

Yes, you can stack multiple User-agent lines before the first Disallow or Allow directive, and all listed agents will be subject to the same rules. For example, placing User-agent: Googlebot and User-agent: Bingbot on consecutive lines, then Disallow: /staging/, applies that Disallow to both crawlers. An empty line ends the block; anything after the empty line starts a new block.

Where should the Sitemap directive go in robots.txt?

The Sitemap directive can appear anywhere in the file, inside or outside a rule block, but convention places it at the bottom, after all User-agent blocks, on its own. It must be followed by an absolute URL (starting with http:// or https://). You can include multiple Sitemap directives if you have a sitemap index or several separate sitemap files. Most major crawlers, including Googlebot and Bingbot, read the Sitemap directive from robots.txt and use it to discover your sitemap. You can also validate the sitemap file itself with our [Sitemap Validator](/sitemap-validator/).

Does robots.txt affect all search engines equally?

No, robots.txt rules are honoured on a per-crawler basis. A Disallow rule for User-agent: Googlebot only applies to Google's crawler; it has no effect on Bingbot or any other crawler. Malicious bots routinely ignore robots.txt entirely, which is another reason the file should not be used for security. Use User-agent: * to set rules for all crawlers at once, and add specific User-agent blocks to override those defaults for particular crawlers.

What is Crawl-delay and is it supported by Google?

Crawl-delay: N tells a crawler to wait N seconds between successive requests to your server. It is designed for servers that cannot handle rapid automated access. Bing, Yandex, and some other crawlers honour Crawl-delay, but Google does not, instead, Google provides a crawl rate setting inside Google Search Console that serves the same purpose. This validator accepts Crawl-delay as a known directive without flagging it as an error.

How do I check if my robots.txt is blocking important pages?

After validating syntax here, use Google Search Console's URL Inspection tool to test specific URLs against your live robots.txt. Enter any URL and the tool will show whether Googlebot can crawl it, which rule in robots.txt is responsible for blocking or allowing it, and whether the page is indexed. For a broad audit, Google Search Console also has a dedicated robots.txt tester under the Legacy Tools section.

Can robots.txt block CSS and JavaScript files?

Technically yes, but it is strongly inadvisable. If you block CSS and JavaScript files, Googlebot cannot render your pages properly and may be unable to understand your site's content or layout. Google's John Mueller has repeatedly advised against blocking these assets. Modern SEO best practice is to allow Googlebot to access all resources needed to render a page, and only block crawlers from pages or sections that should not appear in search results at all.

Also known as

check robots.txtvalidate robots txtrobots txt syntax checkeris my robots.txt validrobots txt testeruser-agent disallow checker