SEO Technical Checklist — robots.txt, Sitemap & Meta Tags

Overview

Technical SEO is the foundation that either enables or undermines everything else — content quality, backlinks, and on-page optimisation are ineffective if your robots.txt blocks key pages, your sitemap contains broken URLs, or your canonical tags point in the wrong direction. These are not difficult problems, but they are invisible ones: a misconfigured robots.txt does not throw an error in your browser, it silently prevents your pages from ranking.

This guide is a sequential checklist for developers launching or auditing a site. The six steps cover the most impactful technical SEO controls in order of importance. Each section includes the specific rules that matter, the tools to implement them correctly, and the validation steps to confirm they are working as intended.

The tools linked in each step are available directly in your browser — you can generate, validate, and test every configuration covered in this guide without installing additional software or logging into an SEO platform.

Step 1: Set Up robots.txt Correctly

The robots.txt file is the first thing most crawlers read when visiting your site. A misconfigured robots.txt can block your entire site from Google's index — or conversely, allow crawlers to waste time on pages that should never be indexed.

Location requirement: robots.txt must be served at https://yourdomain.com/robots.txt — at the root of the domain. A robots.txt at https://yourdomain.com/blog/robots.txt is not recognised by any major search engine. Confirm your robots.txt is accessible without authentication and returns HTTP 200.

Basic structure:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /staging/
Disallow: /?q=
Disallow: /search?

Sitemap: https://yourdomain.com/sitemap.xml

Key rules:

• User-agent: * applies to all crawlers. Add named User-agent: blocks (e.g., User-agent: Googlebot) only if you need crawler-specific rules.
• Allow: / is the default — you only need to declare it if you need to override a broader Disallow.
• The Sitemap: directive is not part of the original robots.txt spec but is widely supported by Google, Bing, and other major crawlers. Always include it.
• Trailing slashes on Disallow: paths block the directory and all pages within it.

What not to block:

• CSS and JavaScript files — Google renders your pages using them. Blocking /static/ or /_next/ prevents Googlebot from seeing your rendered content.
• Images referenced in structured data.
• Any page you want indexed.

Generate and validate:

Use the Robots.txt Generator to build a robots.txt file for your site's structure, then confirm it with the Robots.txt Validator to catch syntax errors, overly broad disallow rules, and missing sitemap declarations before deployment.

After deployment, test specific URLs against your robots.txt using Google Search Console's Robots.txt Tester (under Settings → robots.txt) to confirm that the pages you want crawled are not being blocked.

Step 2: Create and Submit sitemap.xml

An XML sitemap is a structured list of every URL on your site that you want search engines to crawl and index. It does not guarantee indexing, but it significantly improves crawl discovery on new pages and ensures that deep or orphaned pages (without inbound links) are not missed.

What to include:

Only include URLs that are:

• Returning HTTP 200 status
• Not blocked by robots.txt
• Not tagged with noindex meta robots
• Not redirect targets (do not include the redirecting URL — include the final destination)
• Canonical (the preferred version of the page, not a paginated or parameter variant)

What to exclude:

• 301/302 redirects
• 404 and 410 pages
• Pages with noindex meta tag
• Admin, login, and account pages
• Low-quality or thin-content pages you would not want indexed

lastmod accuracy:

The <lastmod> element indicates the last modification date of a page's content. Google uses this to prioritise crawling — pages with recent lastmod dates are recrawled faster. Setting all lastmod values to today's date or leaving them perpetually stale is counterproductive and trains Google to ignore your lastmod signals. Only update lastmod when content actually changes.

Technical limits:

• Maximum 50,000 URLs per sitemap file
• Maximum 50 MB uncompressed (10 MB compressed)
• Use gzip compression for large sitemaps (name the file sitemap.xml.gz)
• For large sites: create multiple sitemap files and reference them from a sitemap index file

Sitemap index format:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
    <lastmod>2026-06-26</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-06-26</lastmod>
  </sitemap>
</sitemapindex>

Generate your sitemap with the Sitemap Generator, then validate it with the Sitemap Validator to confirm format compliance and catch any URLs that are blocked, redirecting, or returning errors.

Submit the sitemap URL to:

• Google Search Console: Property → Sitemaps → Add a new sitemap
• Bing Webmaster Tools: Sitemaps → Submit sitemap
• Also declare in robots.txt via Sitemap: https://yourdomain.com/sitemap.xml

Step 3: Validate All Meta Tags

Meta tags are the primary signal search engines use to understand and represent your pages in search results. Duplicate, missing, or oversized meta tags are among the most common technical SEO issues found in site audits.

Meta title:

• Recommended length: 50–60 characters
• Must be unique across every page on the site
• Should include the primary keyword, ideally near the front
• Brand name is typically appended at the end (e.g., "SIP Calculator | thecalcu.com")
• If truncated in SERP (over ~60 characters), the ellipsis can obscure the most important part

Meta description:

• Recommended length: 150–160 characters
• Must be unique — duplicate descriptions are a quality signal issue
• Should read like a natural, informative sentence, not a keyword list
• Not a direct ranking signal, but strongly influences click-through rate
• Google sometimes rewrites descriptions from page content — having a good description reduces this

Open Graph tags (for social sharing):

<meta property="og:title" content="Page Title Here" />
<meta property="og:description" content="Page description for social sharing." />
<meta property="og:image" content="https://example.com/og-image.jpg" />
<meta property="og:url" content="https://example.com/page/" />
<meta property="og:type" content="website" />

og:image should be 1200×630 pixels minimum for correct display on Facebook and LinkedIn. WhatsApp preview uses a minimum 200×200 pixel image.

Canonical tag:

<link rel="canonical" href="https://example.com/preferred-url/" />

Every page must have a self-referencing canonical, including the canonical page itself. The canonical must:

• Be an absolute URL (including protocol and domain)
• Consistently use https vs http and trailing slash vs no trailing slash to match your chosen convention
• Not point to a different page unless this page's content is genuinely duplicate

Meta robots:

<!-- Default — indexable, links followed -->
<meta name="robots" content="index, follow" />

<!-- Exclude from index but follow links -->
<meta name="robots" content="noindex, follow" />

<!-- Exclude entirely -->
<meta name="robots" content="noindex, nofollow" />

Use noindex on: thank-you pages, admin interfaces, search results pages, paginated pages with thin content, and staging/preview environments.

Validate your meta implementation with the HTML Meta Validator to catch missing, duplicate, or oversized tags across your site's key pages.

Step 4: Fix URL Structure

URL structure affects both crawlability and click-through rate. Clean, descriptive URLs outperform parameter-laden or underscore-based URLs in search results, and they are easier for humans to understand and share.

Correct URL format:

• Lowercase only: /sip-calculator-india/ not /SIP-Calculator/
• Hyphens as word separators: /education-loan-calculator-india/ not /education_loan_calculator/
• Descriptive slugs: /how-to-calculate-sip-returns/ not /article?id=4291
• Short and specific: avoid unnecessary path depth beyond 3 levels
• No parameters for canonical content (UTM and session parameters should not appear in canonical URLs)

Trailing slash consistency:

Choose one convention and enforce it with 301 redirects:

• https://example.com/page/ → canonical (redirect /page → /page/)
• OR https://example.com/page → canonical (redirect /page/ → /page)

Both are valid. Inconsistency is the problem — if both versions return 200, search engines see them as separate duplicate pages.

Common URL issues to fix:

Use the URL Validator to check URL format compliance for your key pages. For larger audits, a crawl tool (Screaming Frog, Ahrefs Site Audit) will catch URL issues at scale.

Redirects: Use 301 (permanent) for all canonical URL changes. A chain of 301 redirects (A→B→C) loses a small amount of PageRank at each hop — resolve chains to direct redirects (A→C) wherever possible.

Step 5: Implement Structured Data (Schema.org)

Structured data is machine-readable markup added to your HTML that helps search engines understand the content and context of your pages. Valid structured data unlocks rich snippets in Google Search — enhanced search result appearances that consistently show higher click-through rates than standard blue-link results.

How to implement: Use JSON-LD (JavaScript Object Notation for Linked Data) embedded in a <script type="application/ld+json"> tag in the page's <head>. JSON-LD is Google's recommended format because it is easier to implement, maintain, and validate than Microdata or RDFa.

High-impact structured data types:

FAQPage — Enables accordion-style FAQ rich results in Google:

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is compound interest?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Compound interest is interest calculated on both the principal and the accumulated interest from previous periods."
      }
    }
  ]
}

HowTo — Enables step-by-step rich results, often with images:

{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "How to Calculate SIP Returns",
  "step": [
    { "@type": "HowToStep", "name": "Enter Monthly Investment", "text": "Enter the amount you plan to invest each month." },
    { "@type": "HowToStep", "name": "Set Expected Return Rate", "text": "Enter the expected annual return rate, typically 10–14% for equity mutual funds." }
  ]
}

BreadcrumbList — Enables breadcrumb display in SERP:

{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://example.com/" },
    { "@type": "ListItem", "position": 2, "name": "Calculators", "item": "https://example.com/calculators/" },
    { "@type": "ListItem", "position": 3, "name": "SIP Calculator", "item": "https://example.com/sip-calculator-india/" }
  ]
}

Testing: Test every structured data implementation with Google's Rich Results Test (search.google.com/test/rich-results) before deploying. A single JSON syntax error or a required field omission can prevent rich snippet eligibility. Schema.org's validator also catches issues that Rich Results Test does not surface.

Step 6: Audit Core Web Vitals

Core Web Vitals are Google's user experience metrics that directly influence search rankings. Pages that fail these thresholds are at a quantifiable disadvantage relative to faster, more stable competitors, particularly when all other ranking signals are equal.

The three metrics and their thresholds:

How to measure: Use Chrome DevTools Lighthouse (audits every metric in a single report), PageSpeed Insights (pagespeed.web.dev — reports both lab and field data from Chrome UX Report), or Google Search Console's Core Web Vitals report (real-user data aggregated from Chrome users).

Common LCP fixes:

• The LCP element is usually a hero image or an H1 heading. Preload it: <link rel="preload" href="hero.webp" as="image" />
• Serve images in WebP or AVIF format — typically 25–50% smaller than equivalent JPEG/PNG
• Use appropriately sized images: a 400px-wide thumbnail should not load a 2000px image
• Enable compression (gzip or Brotli) on all text assets
• Use a CDN to reduce time to first byte (TTFB)

Common CLS fixes:

• Always declare width and height attributes on <img> elements — this reserves layout space before the image loads
• Avoid inserting content above existing content after initial render (banners, late-loading ads above the fold)
• Use CSS aspect-ratio for responsive images
• Ensure web fonts do not cause layout shift: use font-display: optional or preload the font file

Common INP fixes:

• Move long-running JavaScript tasks off the main thread (web workers)
• Break up large JavaScript tasks into smaller chunks using scheduler.yield()
• Defer non-critical third-party scripts (analytics, chat widgets) until after page load
• Audit and remove unused JavaScript — reduce parse and execution time

Monitor Core Web Vitals continuously, not just at launch. A third-party script update, a new ad placement, or a CMS plugin can degrade metrics without any change to your own code.

Key Terms

robots.txt — A plain-text file served at the root of a domain that instructs web crawlers which pages or sections to crawl or avoid, following the Robots Exclusion Standard.

XML Sitemap — A structured XML file listing all URLs a site owner wants search engines to crawl, optionally including last-modified dates and priority hints.

Canonical URL — The preferred, authoritative URL for a page that may be accessible via multiple addresses. Specified via the rel=canonical link element to consolidate ranking signals.

Meta Description — An HTML meta tag providing a brief summary of a page's content, displayed in search engine results pages and social sharing previews.

Structured Data — Machine-readable markup added to web pages using Schema.org vocabulary (typically via JSON-LD) to help search engines understand page content and unlock rich search result features.

JSON-LD (JSON for Linked Data) — A method of encoding structured data as a JSON object embedded in a <script> tag, Google's recommended format for Schema.org markup.

Core Web Vitals — Google's set of user experience metrics (LCP, INP, CLS) used as a ranking signal since 2021, measuring loading performance, interactivity, and visual stability.

LCP (Largest Contentful Paint) — A Core Web Vitals metric measuring the time from page navigation start until the largest visible content element (image or text block) finishes rendering. Should be under 2.5 seconds.

CLS (Cumulative Layout Shift) — A Core Web Vitals metric measuring unexpected visual shifts of page content during load, expressed as a score from 0 to 1. Should be under 0.1.

Crawl Budget — The number of pages a search engine crawler will crawl on a given site within a defined time period, influenced by site health, site authority, and crawl rate settings.