robots.txt Generator

Hand-writing robots.txt is error-prone because the format is stricter than it looks. Forgetting to end a path with / means the rule applies to the file rather than the directory. Using Disallow: with an empty value accidentally allows all paths (it is the correct way to allow everything, but confusingly identical in appearance to a missing value). The generator enforces correct syntax by construction, every Allow and Disallow directive is formatted from your form inputs, not typed manually.

A specific scenario: a developer adding Disallow: /wp-admin without the trailing slash will inadvertently also block /wp-admin-backup/ and any other path beginning with those characters. The correct directive is Disallow: /wp-admin/. The generator appends paths exactly as entered, so your entries should include the trailing slash for directory rules, and a review of the preview output before copying makes any issues immediately visible.

For teams managing multiple sites, a generator also speeds up workflow: spin up a new project, fill in four fields, and have a correct robots.txt ready in under a minute rather than looking up the syntax documentation again.

Website developers and technical SEOs, setting up robots.txt is a standard part of a site launch checklist. This generator produces a correctly formatted file without requiring the developer to memorise directive syntax.

WordPress and CMS site owners, many CMS platforms generate a default robots.txt that may be overly permissive or block certain paths incorrectly. Use this generator to create a custom file that matches your actual site structure.

Content publishers and bloggers, if your blog platform creates duplicate paginated URLs (e.g. /category/tech/page/2/) that you do not want crawled, a custom robots.txt with targeted Disallow rules reduces wasted crawl budget on pages that carry no new content.

AI-conscious content creators, writers and publishers who want to prevent their content from being used for AI model training can add targeted blocks for GPTBot and Google-Extended, both of which officially respect robots.txt. Use this generator to create those blocks correctly alongside the standard Googlebot rules.

E-commerce teams, product filter and sort URLs (e.g. /shoes?colour=red&size=10) generate thousands of near-duplicate pages. Blocking these paths in robots.txt reduces crawl overhead and focuses Google's crawl budget on the canonical category and product pages that actually rank.

The Privacy Policy Generator is a companion tool for site launches, along with robots.txt and a sitemap, a privacy policy is one of the baseline legal documents every website needs.

The generator produces a single output: robots.txt Content, a plain-text file ready to upload to your web server's root directory.

The output shows you exactly which directives will be applied, in the correct order: the User-agent line first, followed by Allow rules (if any), then Disallow rules, then Crawl-delay, then the Sitemap directive. Reading the output before deployment lets you verify that a Disallow rule for /admin/ does not accidentally cover a path you want accessible, and that the user-agent block applies to the right crawler.

If you are blocking a specific crawler like GPTBot, generate a second block by changing the User-agent selection and appending the rules to the same file, the generator shows you the structure for one block at a time, which you can combine manually.

1. Select the User Agent from the dropdown, choose All bots (*) for a global crawl rule, or select a specific crawler (Googlebot, Bingbot, GPTBot) to target just that bot.
2. If you selected Custom, type the exact crawler name in the Custom User Agent Name field, use the crawler's official name as documented by its operator.
3. Enter paths in the Allow Paths text area that should remain accessible even if a broader Disallow rule would otherwise block them, one path per line, starting with /.
4. Enter paths in the Disallow Paths text area that crawlers should skip, use full directory paths ending with / for directories, or exact file paths for individual files.
5. Optionally enter a number in Crawl Delay if you want to slow down non-Googlebot crawlers, note that Googlebot ignores this setting.
6. Enter your sitemap URL in Sitemap URL, this should be the absolute URL to your sitemap.xml file (e.g. https://example.com/sitemap.xml).
7. Copy the generated robots.txt Content from the output box and upload it as a file named robots.txt to the root directory of your web server (the same directory that contains your homepage's index.html).