Short definition
The process of search engine bots discovering and fetching pages.
Beginner In plain English
Search engines send programs called bots that follow links from page to page, reading what they find. That reading is called crawling.
Professional How pros use it
Discovery happens through links and sitemaps. Crawling is limited by robots.txt, server responses and crawl capacity. A page must be crawled before it can be indexed, but crawling does not guarantee indexing.
Examples
- Googlebot following a new link from your homepage.
- A page blocked in robots.txt that cannot be crawled.
My notes
Keep private notes next to anything you read or practice.
Log in to take notes