SEO Technical Checklist — robots.txt, Sitemap & Meta Tags

Overview

Technical SEO is the foundation everything else sits on. Content quality, backlinks, and on-page optimisation don't do much good if your robots.txt blocks key pages, your sitemap contains broken URLs, or your canonical tags point in the wrong direction. None of these problems are hard to fix, but they're invisible: a misconfigured robots.txt won't throw an error in your browser. It just quietly stops your pages from ranking.

This guide works as a sequential checklist for developers launching or auditing a site. The six steps cover the most impactful technical SEO controls in order of importance, and each section includes the specific rules that matter, the tools to put them in place correctly, and the validation steps to confirm they're actually working.

Every tool linked in this guide runs directly in your browser, so you can generate, validate, and test each configuration without installing extra software or logging into an SEO platform.

Step 1: Set Up robots.txt Correctly

The robots.txt file is the first thing most crawlers read when visiting your site. Get it wrong and you can block your entire site from Google's index, or the opposite: let crawlers waste time on pages that should never be indexed at all.

Location requirement: robots.txt must be served at https://yourdomain.com/robots.txt, at the root of the domain. One at https://yourdomain.com/blog/robots.txt won't be recognised by any major search engine. Confirm yours is accessible without authentication and returns HTTP 200.

Basic structure:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /staging/
Disallow: /?q=
Disallow: /search?

Sitemap: https://yourdomain.com/sitemap.xml

Key rules:

• User-agent: * applies to all crawlers. Add named User-agent: blocks (e.g., User-agent: Googlebot) only if you need crawler-specific rules.
• Allow: / is the default. You only need to declare it if you need to override a broader Disallow.
• The Sitemap: directive is not part of the original robots.txt spec but is widely supported by Google, Bing, and other major crawlers. Always include it.
• Trailing slashes on Disallow: paths block the directory and all pages within it.

What not to block:

• CSS and JavaScript files. Google renders your pages using them, and blocking /static/ or /_next/ stops Googlebot from seeing your rendered content.
• Images referenced in structured data.
• Any page you want indexed.

Generate and validate:

Use the Robots.txt Generator to build a robots.txt file for your site's structure, then confirm it with the Robots.txt Validator to catch syntax errors, overly broad disallow rules, and missing sitemap declarations before deployment.

After deployment, test specific URLs against your robots.txt using Google Search Console's Robots.txt Tester (under Settings → robots.txt) to confirm that the pages you want crawled are not being blocked.

Step 2: Create and Submit sitemap.xml

An XML sitemap is a structured list of every URL on your site that you want search engines to crawl and index. It doesn't guarantee indexing, but it does improve crawl discovery on new pages and helps make sure deep or orphaned pages (ones without inbound links) don't get missed.

What to include:

Only include URLs that are:

• Returning HTTP 200 status
• Not blocked by robots.txt
• Not tagged with noindex meta robots
• Not redirect targets (skip the redirecting URL, include the final destination instead)
• Canonical (the preferred version of the page, not a paginated or parameter variant)

What to exclude:

• 301/302 redirects
• 404 and 410 pages
• Pages with noindex meta tag
• Admin, login, and account pages
• Low-quality or thin-content pages you would not want indexed

lastmod accuracy:

The <lastmod> element records the last modification date of a page's content, and Google uses it to prioritise crawling: pages with recent lastmod dates get recrawled faster. Setting every lastmod value to today's date, or leaving them all perpetually stale, backfires, since it trains Google to ignore your lastmod signals. Update it only when content actually changes.

Technical limits:

• Maximum 50,000 URLs per sitemap file
• Maximum 50 MB uncompressed (10 MB compressed)
• Use gzip compression for large sitemaps (name the file sitemap.xml.gz)
• For large sites: create multiple sitemap files and reference them from a sitemap index file

Sitemap index format:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-blog.xml</loc>
    <lastmod>2026-06-26</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-06-26</lastmod>
  </sitemap>
</sitemapindex>

Generate your sitemap with the Sitemap Generator, then validate it with the Sitemap Validator to confirm format compliance and catch any URLs that are blocked, redirecting, or returning errors.

Submit the sitemap URL to:

• Google Search Console: Property → Sitemaps → Add a new sitemap
• Bing Webmaster Tools: Sitemaps → Submit sitemap
• Also declare in robots.txt via Sitemap: https://yourdomain.com/sitemap.xml

Step 3: Validate All Meta Tags

Meta tags are the primary signal search engines use to understand and represent your pages in search results. Duplicate, missing, or oversized meta tags are among the most common technical SEO issues found in site audits.

Meta title:

• Recommended length: 50–60 characters
• Must be unique across every page on the site
• Should include the primary keyword, ideally near the front
• Brand name is typically appended at the end (e.g., "SIP Calculator | thecalcu.com")
• If truncated in SERP (over ~60 characters), the ellipsis can obscure the most important part

Meta description:

• Recommended length: 150–160 characters
• Must be unique; duplicate descriptions are a quality signal issue
• Should read like a natural, informative sentence, not a keyword list
• Not a direct ranking signal, but it strongly influences click-through rate
• Google sometimes rewrites descriptions from page content, and a good one written by you cuts down on that

Open Graph tags (for social sharing):

<meta property="og:title" content="Page Title Here" />
<meta property="og:description" content="Page description for social sharing." />
<meta property="og:image" content="https://example.com/og-image.jpg" />
<meta property="og:url" content="https://example.com/page/" />
<meta property="og:type" content="website" />

og:image should be 1200×630 pixels minimum for correct display on Facebook and LinkedIn. WhatsApp preview uses a minimum 200×200 pixel image.

Canonical tag:

<link rel="canonical" href="https://example.com/preferred-url/" />

Every page must have a self-referencing canonical, including the canonical page itself. The canonical must:

• Be an absolute URL (including protocol and domain)
• Consistently use https vs http and trailing slash vs no trailing slash to match your chosen convention
• Not point to a different page unless this page's content is genuinely duplicate

Meta robots:

<!-- Default: indexable, links followed -->
<meta name="robots" content="index, follow" />

<!-- Exclude from index but follow links -->
<meta name="robots" content="noindex, follow" />

<!-- Exclude entirely -->
<meta name="robots" content="noindex, nofollow" />

Use noindex on: thank-you pages, admin interfaces, search results pages, paginated pages with thin content, and staging/preview environments.

Validate your meta implementation with the HTML Meta Validator to catch missing, duplicate, or oversized tags across your site's key pages.

Step 4: Fix URL Structure

URL structure affects both crawlability and click-through rate. Clean, descriptive URLs outperform parameter-laden or underscore-based URLs in search results, and they are easier for humans to understand and share.

Correct URL format:

• Lowercase only: /sip-calculator-india/ not /SIP-Calculator/
• Hyphens as word separators: /education-loan-calculator-india/ not /education_loan_calculator/
• Descriptive slugs: /how-to-calculate-sip-returns/ not /article?id=4291
• Short and specific: avoid unnecessary path depth beyond 3 levels
• No parameters for canonical content (UTM and session parameters should not appear in canonical URLs)

Trailing slash consistency:

Choose one convention and enforce it with 301 redirects:

• https://example.com/page/ → canonical (redirect /page → /page/)
• OR https://example.com/page → canonical (redirect /page/ → /page)

Both are valid. The problem is inconsistency: if both versions return 200, search engines see them as separate, duplicate pages.

Common URL issues to fix:

Use the URL Validator to check URL format compliance for your key pages. For larger audits, a crawl tool (Screaming Frog, Ahrefs Site Audit) will catch URL issues at scale.

Redirects: Use 301 (permanent) for all canonical URL changes. A chain of 301 redirects (A→B→C) loses a small amount of PageRank at each hop, so resolve chains to direct redirects (A→C) wherever you can.

Step 5: Implement Structured Data (Schema.org)

Structured data is machine-readable markup added to your HTML that helps search engines understand the content and context of your pages. Valid structured data unlocks rich snippets in Google Search, the enhanced search result appearances that consistently pull higher click-through rates than a standard blue link.

How to implement: Use JSON-LD (JavaScript Object Notation for Linked Data) embedded in a <script type="application/ld+json"> tag in the page's <head>. It's Google's recommended format since it's simpler to build and validate than Microdata or RDFa.

High-impact structured data types:

FAQPage: enables accordion-style FAQ rich results in Google.

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is compound interest?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Compound interest is interest calculated on both the principal and the accumulated interest from previous periods."
      }
    }
  ]
}

HowTo: enables step-by-step rich results, often with images.

{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "How to Calculate SIP Returns",
  "step": [
    { "@type": "HowToStep", "name": "Enter Monthly Investment", "text": "Enter the amount you plan to invest each month." },
    { "@type": "HowToStep", "name": "Set Expected Return Rate", "text": "Enter the expected annual return rate, typically 10–14% for equity mutual funds." }
  ]
}

BreadcrumbList: enables breadcrumb display in SERP.

{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://example.com/" },
    { "@type": "ListItem", "position": 2, "name": "Calculators", "item": "https://example.com/calculators/" },
    { "@type": "ListItem", "position": 3, "name": "SIP Calculator", "item": "https://example.com/sip-calculator-india/" }
  ]
}

Testing: Test every structured data implementation with Google's Rich Results Test (search.google.com/test/rich-results) before deploying. A single JSON syntax error or a missing required field can kill rich snippet eligibility. Schema.org's own validator also catches issues that Rich Results Test misses.

Step 6: Audit Core Web Vitals

Core Web Vitals are Google's user experience metrics, and they directly influence search rankings. Pages that miss these thresholds sit at a disadvantage next to faster, more stable competitors when other ranking signals are close to equal.

The three metrics and their thresholds:

How to measure: Use Chrome DevTools Lighthouse (audits every metric in a single report), PageSpeed Insights (pagespeed.web.dev, which reports both lab and field data from Chrome UX Report), or Google Search Console's Core Web Vitals report (real-user data aggregated from Chrome users).

Common LCP fixes:

• The LCP element is usually a hero image or an H1 heading. Preload it: <link rel="preload" href="hero.webp" as="image" />
• Serve images in WebP or AVIF format, typically 25-50% smaller than equivalent JPEG/PNG
• Use appropriately sized images: a 400px-wide thumbnail should not load a 2000px image
• Enable compression (gzip or Brotli) on all text assets
• Use a CDN to cut time to first byte (TTFB)

Common CLS fixes:

• Always declare width and height attributes on <img> elements; this reserves layout space before the image loads
• Avoid inserting content above existing content after initial render (banners, late-loading ads above the fold)
• Use CSS aspect-ratio for responsive images
• Make sure web fonts don't cause layout shift: use font-display: optional or preload the font file

Common INP fixes:

• Move long-running JavaScript tasks off the main thread (web workers)
• Break up large JavaScript tasks into smaller chunks using scheduler.yield()
• Defer non-critical third-party scripts (analytics, chat widgets) until after page load
• Audit and remove unused JavaScript to cut parse and execution time

Keep an eye on Core Web Vitals on an ongoing basis, not just at launch. A third-party script update, a new ad placement, or a CMS plugin can quietly degrade metrics without any change to your own code.

Key Terms

robots.txt: a plain-text file served at the root of a domain that tells web crawlers which pages or sections to crawl or avoid, following the Robots Exclusion Standard.

XML Sitemap: a structured XML file listing all URLs a site owner wants search engines to crawl, optionally including last-modified dates and priority hints.

Canonical URL: the preferred, authoritative URL for a page that may be reachable via multiple addresses, specified via the rel=canonical link element to consolidate ranking signals.

Meta Description: an HTML meta tag providing a brief summary of a page's content, displayed in search engine results pages and social sharing previews.

Structured Data: machine-readable markup added to web pages using Schema.org vocabulary (typically via JSON-LD) so search engines can understand page content and unlock rich search result features.

JSON-LD (JSON for Linked Data): a method of encoding structured data as a JSON object embedded in a <script> tag, Google's recommended format for Schema.org markup.

Core Web Vitals: Google's set of user experience metrics (LCP, INP, CLS) used as a ranking signal since 2021, measuring loading performance, interactivity, and visual stability.

LCP (Largest Contentful Paint): a Core Web Vitals metric measuring the time from page navigation start until the largest visible content element (image or text block) finishes rendering. Should stay under 2.5 seconds.

CLS (Cumulative Layout Shift): a Core Web Vitals metric measuring unexpected visual shifts of page content during load, expressed as a score from 0 to 1. Should stay under 0.1.

Crawl Budget: the number of pages a search engine crawler will crawl on a given site within a defined time period, shaped by site health, site authority, and crawl rate settings.