Home›Generators›Developer Tools›robots.txt Generator

robots.txt Generator

Developer Tools

Generate a valid robots.txt file for your website. Control crawler access per user-agent, set allow and disallow paths, and add your sitemap URL — free.

What is a robots.txt?

Every website has the option to place a robots.txt file at its root — at the path /robots.txt — to communicate crawling preferences to search engine bots and other automated agents. The Robots Exclusion Protocol, which governs this file, is an informal but widely respected standard followed by Googlebot, Bingbot, DuckDuckBot, GPTBot, and dozens of other compliant crawlers.

A robots.txt Generator creates this file for you from a form interface, eliminating the need to hand-write the directives and check them for syntax errors. The file format is simple but unforgiving: a single misplaced space or an incorrect path prefix can accidentally block your entire website from Google — a mistake that can take weeks to reverse once search engines process the change.

The file consists of one or more rule blocks. Each block opens with a User-agent: line identifying the targeted crawler, followed by Allow: or Disallow: directives for individual paths or directories. An optional Crawl-delay: directive controls how fast a crawler may request pages. A Sitemap: directive at the end points crawlers to your XML sitemap.

Understanding what robots.txt does and does not do is critical: it controls whether a crawler visits a URL, but it does not control whether that URL appears in search results. A page can be indexed by Google without ever being crawled, if enough other sites link to it. For true de-indexing you need a noindex meta tag on the page itself.

Common legitimate uses for robots.txt include:

  • Blocking crawlers from /admin/, /checkout/, or other non-public paths that waste crawl budget
  • Preventing duplicate content from being crawled (e.g. filtered search result URLs like /products?sort=price)
  • Blocking AI training crawlers (GPTBot, Google-Extended) from proprietary content
  • Pointing all crawlers to your sitemap via the Sitemap: directive

Pair this tool with Sitemap.xml Generator to complete the two foundational technical SEO files every website needs, and use UTM Link Generator to make sure traffic that does reach your site is properly attributed in your analytics platform.

How to use this robots.txt calculator

  1. Select the User Agent from the dropdown — choose All bots (*) for a global crawl rule, or select a specific crawler (Googlebot, Bingbot, GPTBot) to target just that bot.
  2. If you selected Custom, type the exact crawler name in the Custom User Agent Name field — use the crawler's official name as documented by its operator.
  3. Enter paths in the Allow Paths text area that should remain accessible even if a broader Disallow rule would otherwise block them — one path per line, starting with /.
  4. Enter paths in the Disallow Paths text area that crawlers should skip — use full directory paths ending with / for directories, or exact file paths for individual files.
  5. Optionally enter a number in Crawl Delay if you want to slow down non-Googlebot crawlers — note that Googlebot ignores this setting.
  6. Enter your sitemap URL in Sitemap URL — this should be the absolute URL to your sitemap.xml file (e.g. https://example.com/sitemap.xml).
  7. Copy the generated robots.txt Content from the output box and upload it as a file named robots.txt to the root directory of your web server (the same directory that contains your homepage's index.html).

Formula & Methodology

The robots.txt file follows the Robots Exclusion Protocol. The generator assembles directives in this order:

User-agent: [agent] Allow: [path]          ← zero or more, before Disallow rules Disallow: [path]       ← zero or more Crawl-delay: [n]       ← optional                        ← blank line between blocks Sitemap: [url]         ← outside any user-agent block, at end of file

Path matching rules:
- Disallow: /admin/ — blocks the /admin/ directory and everything under it
- Disallow: /admin — blocks any URL starting with /admin (including /admin-panel/)
- Disallow: /*.pdf$ — blocks all PDF files (supported by Google, not all crawlers)
- Allow: / with no Disallow — permits all paths (equivalent to no robots.txt for that agent)
- Disallow: / — blocks all paths for the specified agent

Example output for a typical site:

User-agent: * Allow: / Disallow: /admin/ Disallow: /checkout/ Disallow: /account/ Crawl-delay: 1  Sitemap: https://example.com/sitemap.xml

This allows all bots to crawl everything except the three protected directories, asks non-Google crawlers to wait one second between requests, and points every crawler to the sitemap.

Frequently Asked Questions

A robots.txt file is a plain-text file placed at the root of a website (e.g. https://example.com/robots.txt) that instructs web crawlers which pages or directories they should or should not access. It follows the Robots Exclusion Protocol, a widely respected web standard. Crawlers from Google, Bing, and other search engines check this file before crawling your site to understand your crawling preferences.
No — robots.txt controls crawler access, not indexing. Disallowing a URL in robots.txt tells the crawler not to visit that URL, but if other pages link to it, search engines may still index it without ever crawling the page content. To prevent a page from appearing in search results entirely, use a 'noindex' meta tag or X-Robots-Tag HTTP header on the page itself, not robots.txt.
Disallow tells the specified user-agent not to crawl the listed path or directory. Allow explicitly permits a specific path that would otherwise be blocked by a broader Disallow rule. For example, you might Disallow /images/ to block image crawling but Allow /images/logo.png to let crawlers access just the logo. Without an Allow override, the more specific Disallow rule applies.
The asterisk (*) is a wildcard that matches all crawlers — Googlebot, Bingbot, GPTBot, and any other compliant bot. A rule block starting with 'User-agent: *' applies to every crawler that reads the file. You can have multiple blocks in a single robots.txt file, each targeting a different user-agent. Googlebot-specific rules override the wildcard rules for Google's crawler.
Crawl-delay instructs a crawler to wait a specified number of seconds between successive requests to your server. It is useful for shared hosting or resource-limited servers where aggressive crawling could cause performance issues. Note that Google officially does not respect Crawl-delay in robots.txt — use Google Search Console's crawl rate settings for Googlebot. Bingbot and some other crawlers do honour it.
Yes — this is considered best practice. Adding a Sitemap directive (e.g. 'Sitemap: https://example.com/sitemap.xml') tells all crawlers where to find your XML sitemap, regardless of whether they have been submitted via Search Console or Bing Webmaster Tools. Some crawlers that are not registered in any webmaster tool will discover your sitemap solely through the robots.txt Sitemap directive.
Yes. You can have as many user-agent blocks as needed, each with its own Allow and Disallow rules. Place each block separately with a blank line between them. For example, you might allow all paths for Googlebot while blocking /ai-content/ for GPTBot to prevent AI training crawlers from accessing your proprietary content. Rules in one block do not affect another user-agent's block.
No. robots.txt is a public file that any person or malicious bot can read. Disallowing a path does not make it inaccessible — it simply asks compliant crawlers not to visit it. Never rely on robots.txt to hide sensitive pages like /admin/ or API endpoints. Use proper authentication, server-level access controls, and firewalls for security. The Disallow rule is an SEO and crawl-budget tool, not a security measure.
Select the user-agent you want to target from the dropdown — choose 'All bots (*)' for a global rule, or pick a specific crawler like Googlebot or GPTBot. Enter your allowed and disallowed paths in the respective text areas, one path per line. Optionally add a crawl delay and your sitemap URL, then copy the generated code from the output box and upload it as robots.txt to the root directory of your website.
Yes, for compliant AI crawlers. OpenAI's GPTBot, Google's Google-Extended, and Common Crawl's CCBot all respect robots.txt. Add a separate User-agent block for each crawler you want to block and set Disallow: / to block all paths. However, not all AI data-collection bots respect robots.txt — those that ignore the standard cannot be stopped this way. Major AI companies have publicly committed to honouring robots.txt restrictions.
No. The robots.txt Generator runs entirely in your browser. Nothing you enter — including your site paths, crawl delay, or sitemap URL — is transmitted to any server. All processing happens locally in your browser, so you can safely use your real production URLs without any privacy concern.
Also known as
robots txt file generatorSEO crawl controlblock googlebotdisallow robotsrobots.txt builderwebsite crawler rules