Home›Validators›Data›Robots.txt Validator

Robots.txt Validator

Data

Paste your robots.txt and instantly validate its syntax. Checks User-agent blocks, Disallow/Allow paths, Sitemap URLs, and unknown directives. Free, no signup.

What is a Robots.txt?

A robots.txt validator checks whether your robots.txt file is correctly written according to the Robots Exclusion Protocol — the standard that governs how search engine crawlers interact with websites. A single syntax error in robots.txt can silently block an entire section of your website from being crawled, or fail to protect the pages you intended to hide from search engines.

The robots.txt file lives at the root of your domain — always at https://yoursite.com/robots.txt — and is one of the first files a search engine crawler reads when it visits your site. It is plain text, not XML or JSON, so there is no schema for parsers to validate against automatically. That means malformed directives are simply ignored by crawlers rather than throwing an error, making it easy to deploy a broken robots.txt without realising it for days or weeks.

This validator reads your robots.txt line by line, applying the same parsing rules that major crawlers use. It checks that every Disallow and Allow directive is preceded by a User-agent block, that path values begin with a forward slash, that Sitemap directives reference absolute URLs, and that no unrecognised directives are present. It also reports a summary — number of rule blocks, total Disallow and Allow rules, and Sitemap references — so you can confirm the file's shape matches your intentions.

For the corresponding sitemap file referenced by your robots.txt, our Sitemap Validator checks the XML structure and URL count. For individual URL format checks, the URL Validator is the right tool.

How to use this Robots.txt calculator

  1. Get your robots.txt content — open https://yoursite.com/robots.txt in a browser, select all the text, and copy it. Alternatively, open the file directly from your server or CMS.

  2. Paste it into the input field — the Robots.txt Content textarea accepts the raw text exactly as it appears in the file. There is no need to wrap it in quotes or modify the formatting.

  3. Read the result — validation runs automatically as you type or paste. A green Valid badge means the file is syntactically correct; a red Invalid badge is followed by a list of specific errors with line numbers.

  4. Fix any errors — each error message includes the line number and a description of what is wrong. Return to your file, fix the indicated line, and paste the updated version to re-validate.

  5. Check the summary stats — even on a valid file, review the rule block count and Disallow count to confirm they match your expectations. An unexpectedly high Disallow count or a missing Sitemap directive can indicate a misconfiguration that is technically valid but not what you intended.

  6. Validate your sitemap — if the file includes one or more Sitemap: directives, paste the sitemap XML into our Sitemap Validator to confirm the referenced file is also well-formed.

Frequently Asked Questions

A robots.txt file is a plain-text file placed at the root of a website (https://yoursite.com/robots.txt) that tells search engine crawlers which parts of the site they are and are not allowed to access. It follows the Robots Exclusion Protocol, a de facto standard supported by Google, Bing, and most other major crawlers. The file is publicly readable — anyone can view it — so it should never be used to protect sensitive content, only to manage crawler behaviour.
This validator checks that your robots.txt file follows the Robots Exclusion Protocol correctly. It verifies that every Disallow and Allow directive is preceded by a User-agent directive, that path values start with a forward slash, that Sitemap URLs are absolute http(s) addresses, and that no unknown or misspelled directives are present. It also counts rule blocks, Disallow rules, Allow rules, and Sitemap entries so you can audit the file's coverage at a glance.
Disallow tells a crawler not to access the specified path or any URL that begins with that path. Allow is used to create exceptions within a Disallow rule — for example, you could Disallow an entire directory but Allow a specific file inside it. Most major crawlers, including Googlebot, honour both directives. When Allow and Disallow rules conflict for the same URL, the more specific (longer) path takes precedence; if both are the same length, the Allow wins.
Disallow: / tells crawlers not to crawl any page on your site, but it does not prevent pages from being indexed if they are linked to from other websites. Google can index a URL it has never crawled if it appears in an external link, showing a result without any snippet. To prevent indexing entirely, you need a noindex meta tag or HTTP header on the page itself — but that requires the page to be crawlable in the first place, which creates a catch-22 for blocked pages.
User-agent: * applies the following rules to all crawlers that are not covered by a more specific User-agent block. It acts as a catch-all. Most robots.txt files include at least one User-agent: * block to set a baseline policy for any crawler that visits the site. If you want to allow all crawlers unrestricted access, you still need the block — use User-agent: * followed by Disallow: (empty value) to explicitly permit everything.
An empty Disallow: line (nothing after the colon) means the crawler is allowed to access the entire site. It is the conventional way to tell a search engine that it may crawl everything — equivalent to having no Disallow rules at all. Many CMS platforms generate this as the default robots.txt for public sites: User-agent: * on one line, then Disallow: on the next.
Yes — you can stack multiple User-agent lines before the first Disallow or Allow directive, and all listed agents will be subject to the same rules. For example, placing User-agent: Googlebot and User-agent: Bingbot on consecutive lines, then Disallow: /staging/, applies that Disallow to both crawlers. An empty line ends the block; anything after the empty line starts a new block.
The Sitemap directive can appear anywhere in the file — inside or outside a rule block — but convention places it at the bottom, after all User-agent blocks, on its own. It must be followed by an absolute URL (starting with http:// or https://). You can include multiple Sitemap directives if you have a sitemap index or several separate sitemap files. Most major crawlers, including Googlebot and Bingbot, read the Sitemap directive from robots.txt and use it to discover your sitemap. You can also validate the sitemap file itself with our [Sitemap Validator](/sitemap-validator/).
No — robots.txt rules are honoured on a per-crawler basis. A Disallow rule for User-agent: Googlebot only applies to Google's crawler; it has no effect on Bingbot or any other crawler. Malicious bots routinely ignore robots.txt entirely, which is another reason the file should not be used for security. Use User-agent: * to set rules for all crawlers at once, and add specific User-agent blocks to override those defaults for particular crawlers.
Crawl-delay: N tells a crawler to wait N seconds between successive requests to your server. It is designed for servers that cannot handle rapid automated access. Bing, Yandex, and some other crawlers honour Crawl-delay, but Google does not — instead, Google provides a crawl rate setting inside Google Search Console that serves the same purpose. This validator accepts Crawl-delay as a known directive without flagging it as an error.
After validating syntax here, use Google Search Console's URL Inspection tool to test specific URLs against your live robots.txt. Enter any URL and the tool will show whether Googlebot can crawl it, which rule in robots.txt is responsible for blocking or allowing it, and whether the page is indexed. For a broad audit, Google Search Console also has a dedicated robots.txt tester under the Legacy Tools section.
Technically yes, but it is strongly inadvisable. If you block CSS and JavaScript files, Googlebot cannot render your pages properly and may be unable to understand your site's content or layout. Google's John Mueller has repeatedly advised against blocking these assets. Modern SEO best practice is to allow Googlebot to access all resources needed to render a page, and only block crawlers from pages or sections that should not appear in search results at all.
Also known as
check robots.txtvalidate robots txtrobots txt syntax checkeris my robots.txt validrobots txt testeruser-agent disallow checker