Video lesson
How Search Engines Crawl, Index and Rank Pages
A short video lesson on the three stages every page goes through before it can appear in search, with the full transcript.
Video
The video for this lesson is on its way. The full lesson is written out below, so you can learn it now.
Key takeaways
- Pages go through crawling, indexing and ranking, in that order.
- Diagnose problems by checking each stage in turn.
- Search Console’s page indexing report explains most indexing issues.
Lesson transcript
Every page you work on goes through three stages before anyone can find it in search: crawling, indexing and ranking. Knowing which stage a problem belongs to tells you where to look.
1. Crawling
Search engines use bots that follow links and read sitemaps to discover pages. If a bot cannot reach a page, because it is blocked by robots.txt, returns an error, or nothing links to it, the process stops here.
2. Indexing
After fetching a page, the search engine decides whether to store it. Pages can be left out because of a noindex tag, because a canonical tag points to another URL, or because the content duplicates another page. Search Console’s page indexing report shows the reason.
3. Ranking
When someone searches, the engine picks the most relevant, helpful indexed pages and orders them. Relevance to the search intent, content quality, links and page experience all play a part.
What this means for your work
When a client says “my page isn’t on Google,” check the stages in order: can it be crawled, is it indexed, and only then, why does it rank where it does.
Terms used here
Practice what you learned
- Assignment Audit a small online store for crawl errors Use a crawl export to find broken links, redirect chains, and indexing problems, then write a…
My notes
Keep private notes next to anything you read or practice.
Log in to take notes