A robots.txt file is one of the simplest technical SEO files you will ever write — and one of the most consequential to get wrong. A single typo can block your entire site from Google's index. This guide covers the syntax, the key directives, real-world examples for common setups, and how to verify the file before it goes live.
Overview
The Robots Exclusion Protocol is a standard created in 1994 that defines how web crawlers should request permission before accessing parts of a website. Your robots.txt file implements this protocol. It is fetched by every compliant crawler before they begin crawling your domain.
robots.txt controls crawl access, not security and not indexing. It is advisory — good bots follow it, bad bots ignore it. Understanding this distinction prevents a common mistake: treating robots.txt as a privacy tool for sensitive content.
You can generate a robots.txt file using the Robots.txt Generator and validate an existing one using the Robots.txt Validator.
What You Need
- Text editor (robots.txt is plain text — no special tools required)
- A clear list of which paths should be blocked and which bots you want to target
- FTP/SFTP access or a deployment workflow that can place a file at your domain root
- Access to Google Search Console for testing (recommended before going live)
Step 1: Understand What robots.txt Does (and Doesn't Do)
Before writing a single line, get these facts straight:
robots.txt controls crawling, not indexing:
- Blocking a URL in robots.txt → Google will not visit it
- But Google can still index the URL from link signals alone — showing it in results with no title or description
- To prevent indexing: use
<meta name="robots" content="noindex">on the page itself
robots.txt is advisory, not enforced:
- Googlebot, Bingbot, and well-behaved crawlers respect it
- Malicious bots, scrapers, and scanners do not — treat robots.txt as a crawl guide, never as a security boundary
File location is fixed:
- Must be at
https://yourdomain.com/robots.txt— the root of the domain - Works per-domain, not per-subdirectory
- Subdomains each need their own robots.txt:
https://blog.example.com/robots.txt
Path matching is case-sensitive on Linux servers, which host the vast majority of websites.
Step 2: Start with the Basic Syntax
robots.txt uses a simple plain-text format with three types of lines:
User-agent: [bot name or *]
Disallow: [path to block]
Allow: [path to explicitly permit]
Rules of the format:
User-agentspecifies which crawler the following rules apply to. Use*for all crawlers.Disallowwith a path blocks access to that path and everything under it.Disallow:with nothing (blank value) means "allow everything" — used to explicitly allow a bot.Allowoverrides a Disallow for a more specific path.- A blank line separates different records (different User-agent groups).
- Lines beginning with
#are comments and are ignored by crawlers.
Minimal valid robots.txt (allow all crawlers everywhere):
User-agent: *
Disallow:
Minimal valid robots.txt (block all crawlers from everything — use with extreme caution):
User-agent: *
Disallow: /
The second example blocks your entire site. This is the correct form for a staging server. It is a catastrophe if deployed to production.
Step 3: Write Common Directives
Here are the most frequently used patterns with explanations:
Block admin and private areas:
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/
Disallow: /login/
Disallow: /dashboard/
Block URL parameters that create duplicate content (e.g., session IDs, sort filters):
User-agent: *
Disallow: /*?sessionid=
Disallow: /*?sort=
Disallow: /*?ref=
Allow everything except one directory:
User-agent: *
Allow: /
Disallow: /staging/
Block a specific SEO crawler (e.g., AhrefsBot):
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: *
Allow: /
Allow Googlebot while blocking everything else:
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /
Add your sitemap (always recommended):
User-agent: *
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
The Sitemap directive is not part of the original RFC but is supported by Google, Bing, and most major crawlers. Place it at the end of the file. If you have multiple sitemaps, list each on its own Sitemap: line.
Using Allow to override a broader Disallow:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
This blocks the WordPress admin directory but explicitly permits the admin-ajax.php endpoint, which is used for front-end AJAX requests and must remain accessible for certain plugins to function.
Step 4: Test Before Deploying
Never push a robots.txt file to production without testing it.
Google Search Console robots.txt tester:
- Log into Google Search Console
- Navigate to Settings → robots.txt
- Paste your robots.txt content
- Enter a URL you want to test (e.g.,
/admin/login) - Select the User-agent (e.g., Googlebot)
- The tester shows whether that URL is Allowed or Blocked
Online validator: Use the Robots.txt Validator to check syntax and simulate which paths would be blocked for any given bot.
Manual inspection: Fetch your live robots.txt at https://yourdomain.com/robots.txt in a browser. Confirm the file serves as plain text (Content-Type: text/plain) and that the content matches what you expect. Cached or CDN-served versions can sometimes serve outdated files.
Test the critical paths explicitly:
- Your homepage: should be Allowed
- Your blog or product pages: should be Allowed
- Your admin path: should be Blocked
- Your sitemap URL: should be Allowed (even if the sitemap directory is blocked elsewhere)
Step 5: Common CMS Configurations
WordPress:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Disallow: /?s=
Disallow: /search/
Disallow: /tag/
Disallow: /author/
Sitemap: https://example.com/sitemap_index.xml
Block the WordPress admin, core includes, internal search results, and tag/author archives (which often create thin duplicate content). The ?s= parameter is WordPress's search query string.
Next.js static export (like thecalcu.com):
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
For a clean static export, typically all routes are worth indexing. The _next/static/ directory serves JavaScript and CSS bundles — these don't need to be crawled but are harmless to leave open. Block only if you have specific paths to hide (staging routes, internal tool admin pages).
Shopify:
Shopify automatically generates a robots.txt file and does not allow full replacement (as of 2024), but allows appending custom rules via the theme. Common additions block /collections/*+* (faceted navigation URLs that create duplicates) and /products/*?variant=.
Common Mistakes to Avoid
Accidentally blocking the entire site (Disallow: /): The single most dangerous mistake. This is the correct form for staging servers — deploying it to production has caused high-profile SEO disasters. Always sanity-check this line before deploying. Some CI/CD pipelines that pull robots.txt from environment variables have deployed staging configs to production.
Using robots.txt to protect sensitive data: robots.txt is public. Your /admin/ path, /private/ directory, and any other blocked paths are explicitly listed in a file that anyone can read at yourdomain.com/robots.txt. Blocking these paths from crawlers is correct; thinking it hides them from humans is wrong. Use authentication to protect sensitive areas.
Forgetting the Sitemap directive: Many site owners configure robots.txt once and forget to add the Sitemap line when they later generate a sitemap. Without it, crawlers have to discover your sitemap via Search Console alone. Adding Sitemap: to robots.txt is a passive, always-on notification to any crawler.
Incorrect case sensitivity: On Linux servers, /Admin/ and /admin/ are different paths. If your CMS uses /Admin/ (capital A), your Disallow: /admin/ rule does nothing. Always match the exact case of your actual URL paths.
Blocking CSS and JavaScript files: Blocking /assets/, /static/, or other resource directories prevents Googlebot from rendering your pages correctly. Google uses JavaScript rendering and needs access to CSS and JS to fully understand page content and layout. Blocking these files can cause your pages to be misclassified in search.
Formula & Methodology
The robots.txt matching algorithm (as implemented by Google) works as follows:
- The crawler fetches
[protocol]://[host]/robots.txt - It finds all matching User-agent groups (exact name match, then
*fallback) - For the matching group(s), it evaluates all Disallow and Allow directives against the URL path
- The longest matching directive wins, regardless of whether it is Allow or Disallow
- If no directive matches, the URL is allowed
Wildcard path matching:
*matches any sequence of characters$matches the end of the URLDisallow: /*.pdf$→ blocks all .pdf URLsDisallow: /search*→ blocks /search, /search?q=foo, /search/results
Crawl-delay (unofficial):
User-agent: Bingbot
Crawl-delay: 5
This asks Bing to wait 5 seconds between requests. Google does not respect this directive — manage Googlebot's crawl rate in Search Console under Settings → Crawl stats → Open Crawl Rate Settings.
The authoritative reference for Googlebot's interpretation is the Google Robots.txt specification, which extends the original 1994 protocol with wildcard support. For Bing, see the Bing Webmaster Guidelines.