A robots.txt file is one of the simplest technical SEO files you'll ever write, and one of the most consequential to get wrong. A single typo can block your entire site from Google's index. This guide covers the syntax, the key directives, real-world examples for common setups, and how to verify the file before it goes live.
Overview
The Robots Exclusion Protocol is a standard from 1994 that defines how web crawlers should request permission before accessing parts of a website. Your robots.txt file implements this protocol, and every compliant crawler fetches it before crawling your domain at all.
robots.txt controls crawl access, not security and not indexing. It's advisory: good bots follow it, bad bots ignore it. Keeping that distinction straight prevents a common mistake, treating robots.txt as a privacy tool for sensitive content.
You can generate a robots.txt file using the Robots.txt Generator and validate an existing one using the Robots.txt Validator.
What You Need
- Text editor (robots.txt is plain text, no special tools required)
- A clear list of which paths should be blocked and which bots you want to target
- FTP/SFTP access or a deployment workflow that can place a file at your domain root
- Access to Google Search Console for testing (recommended before going live)
Step 1: Understand What robots.txt Does (and Doesn't Do)
Before writing a single line, get these facts straight.
robots.txt controls crawling, not indexing. Blocking a URL means Google won't visit it, but Google can still index that URL from link signals alone, showing it in results with no title or description. To actually prevent indexing, use <meta name="robots" content="noindex"> on the page itself.
robots.txt is advisory, not enforced. Googlebot, Bingbot, and other well-behaved crawlers respect it. Malicious bots, scrapers, and scanners don't, so treat robots.txt as a crawl guide, never as a security boundary.
The file location is fixed. It must sit at https://yourdomain.com/robots.txt, the root of the domain, and it works per-domain, not per-subdirectory. Subdomains each need their own robots.txt, like https://blog.example.com/robots.txt.
Path matching is case-sensitive on Linux servers, which host most websites in production.
Step 2: Start with the Basic Syntax
robots.txt uses a simple plain-text format with three types of lines:
User-agent: [bot name or *]
Disallow: [path to block]
Allow: [path to explicitly permit]
A few rules govern the format. User-agent specifies which crawler the following rules apply to, and * covers all crawlers. Disallow with a path blocks access to that path and everything under it. Disallow: with nothing after it means "allow everything," which is how you explicitly permit a bot. Allow overrides a Disallow for a more specific path. A blank line separates different records (different User-agent groups), and lines starting with # are comments crawlers ignore.
Here's a minimal valid robots.txt that allows all crawlers everywhere:
User-agent: *
Disallow:
And one that blocks all crawlers from everything, use with extreme caution:
User-agent: *
Disallow: /
The second example blocks your entire site. It's the right form for a staging server. It's a catastrophe if it ends up in production.
Step 3: Write Common Directives
Here are the patterns you'll reach for most often.
Block admin and private areas:
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/
Disallow: /login/
Disallow: /dashboard/
Block URL parameters that create duplicate content, like session IDs or sort filters:
User-agent: *
Disallow: /*?sessionid=
Disallow: /*?sort=
Disallow: /*?ref=
Allow everything except one directory:
User-agent: *
Allow: /
Disallow: /staging/
Block a specific SEO crawler, such as AhrefsBot:
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: *
Allow: /
Allow Googlebot while blocking everything else:
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
Add your sitemap, which is always worth doing:
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
The Sitemap directive isn't part of the original RFC, but Google, Bing, and most major crawlers support it. Place it at the end of the file, and if you have multiple sitemaps, list each on its own Sitemap: line.
Using Allow to override a broader Disallow looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
This blocks the WordPress admin directory but explicitly permits the admin-ajax.php endpoint, which handles front-end AJAX requests and needs to stay accessible for certain plugins to work.
Step 4: Test Before Deploying
Never push a robots.txt file to production without testing it first.
For Google Search Console's robots.txt tester: log into Search Console, navigate to Settings then robots.txt, paste your robots.txt content, enter a URL you want to test (like /admin/login), select the User-agent (Googlebot, say), and the tester shows whether that URL is Allowed or Blocked.
Or use an online validator: the Robots.txt Validator checks syntax and simulates which paths would be blocked for any given bot.
You can also inspect manually. Fetch your live robots.txt at https://yourdomain.com/robots.txt in a browser. Confirm it serves as plain text (Content-Type: text/plain) and that the content matches what you expect, since cached or CDN-served versions sometimes serve outdated files.
Test the critical paths explicitly. Your homepage should be Allowed. Blog or product pages should be Allowed. Your admin path should be Blocked. Your sitemap URL should be Allowed, even if the sitemap directory itself is blocked elsewhere.
Step 5: Common CMS Configurations
For WordPress:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /?s=
Disallow: /search/
Disallow: /tag/
Disallow: /author/
Sitemap: https://example.com/sitemap_index.xml
This blocks the WordPress admin, core includes, internal search results, and tag/author archives, which often create thin duplicate content. The ?s= parameter is WordPress's search query string.
For a Next.js static export (like thecalcu.com):
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
For a clean static export, most routes are worth indexing. The _next/static/ directory serves JavaScript and CSS bundles that don't need crawling but are harmless to leave open. Block only the paths you specifically need hidden, like staging routes or internal tool admin pages.
For Shopify: it automatically generates a robots.txt file and doesn't allow full replacement (as of 2024), but it does allow appending custom rules via the theme. Common additions block /collections/*+* (faceted navigation URLs that create duplicates) and /products/*?variant=.
Common Mistakes to Avoid
Accidentally blocking the entire site (Disallow: /). This is the single most dangerous mistake to make. It's the correct form for staging servers, but deploying it to production has caused high-profile SEO disasters. Sanity-check this line every time before deploying. Some CI/CD pipelines that pull robots.txt from environment variables have deployed staging configs to production by mistake.
Using robots.txt to protect sensitive data. robots.txt is public. Your /admin/ path, /private/ directory, and any other blocked paths sit listed in a file anyone can read at yourdomain.com/robots.txt. Blocking these paths from crawlers makes sense; assuming it hides them from humans doesn't. Use authentication to protect sensitive areas instead.
Forgetting the Sitemap directive. Plenty of site owners set up robots.txt once and forget to add the Sitemap line when they later generate a sitemap. Without it, crawlers have to discover your sitemap through Search Console alone. Adding Sitemap: to robots.txt is a passive, always-on notification to any crawler that visits.
Incorrect case sensitivity. On Linux servers, /Admin/ and /admin/ are different paths. If your CMS uses /Admin/ with a capital A, your Disallow: /admin/ rule does nothing at all. Match the exact case of your actual URL paths.
Blocking CSS and JavaScript files. Blocking /assets/, /static/, or other resource directories stops Googlebot from rendering your pages correctly. Google renders JavaScript and needs access to CSS and JS to understand page content and layout fully, so blocking these files can get your pages misclassified in search.
Formula & Methodology
The robots.txt matching algorithm, as Google implements it, works like this:
- The crawler fetches
[protocol]://[host]/robots.txt - It finds all matching User-agent groups (exact name match, then
*fallback) - For the matching group(s), it evaluates all Disallow and Allow directives against the URL path
- The longest matching directive wins, regardless of whether it's Allow or Disallow
- If nothing matches, the URL is allowed
For wildcard path matching: * matches any sequence of characters, and $ matches the end of the URL. Disallow: /*.pdf$ blocks all .pdf URLs, and Disallow: /search* blocks /search, /search?q=foo, and /search/results.
Crawl-delay, unofficial as it is, looks like this:
User-agent: Bingbot
Crawl-delay: 5
This asks Bing to wait 5 seconds between requests. Google doesn't respect this directive at all; manage Googlebot's crawl rate in Search Console under Settings, then Crawl stats, then Open Crawl Rate Settings.
The authoritative reference for Googlebot's interpretation is the Google Robots.txt specification, which extends the original 1994 protocol with wildcard support. For Bing, check the Bing Webmaster Guidelines.