Standard operating procedure

SOP: Technical SEO Crawl Check

Crawl a site, find the issues that block crawling and indexing, and turn them into a prioritized fix list.

SOP7 stepsAbout 150 minAdvanced

Objective

Make sure search engines can reach, read and index the pages that matter, and remove what wastes their time.

When to use

  • Quarterly for active clients.
  • Before and after a redesign or migration.
  • When important pages disappear from search.

Prerequisites

  • A site crawler (desktop or cloud).
  • Access to Google Search Console.
  • The Technical SEO Template.

Steps

  1. Crawl the site

    Crawl from the homepage and export all HTML URLs with status codes.

  2. Find broken pages

    List 4xx errors and the pages that link to them.

  3. Check redirects

    Flag redirect chains and redirects that point to irrelevant pages.

  4. Check indexability

    Look for noindex tags, robots.txt blocks and canonicals pointing elsewhere.

  5. Compare with the sitemap

    Indexable pages should be listed; redirects and errors should not.

  6. Review Search Console

    Check the page indexing report for important excluded pages.

  7. Prioritize fixes

    Start with issues on high-value pages, then sitewide patterns.

Quality checklist

0 of 5 checked

Run through these before you hand the work over. Your ticks are saved in this browser only.

Download

A plain-text Markdown copy (.md) that opens in any editor and pastes cleanly into Google Docs, Notion or ClickUp.

Download SOP (Markdown)

Templates for this SOP

  • Template Technical SEO Template Record crawl and indexing checks URL by URL: status codes, indexability, canonicals, sitemap coverage and fixes.
  • Template SEO Audit Template Log every audit finding with evidence, severity and a recommended fix, plus a summary tab that…

Practice it

My notes

Keep private notes next to anything you read or practice.

Log in to take notes