Web crawler: what is it and how does it work?

What is a web crawler?

A web crawler is a program that moves across the web from link to link to discover pages and download them. Google uses the generic term “crawler”, or simply “robot”, for any program that automatically discovers and scans websites.

Common synonyms: spider, bot, robot, web robot. “Indexing robot” is also heard, but the term is inaccurate: the robot crawls, and indexing is a separate step at Google.

Crawling is the first step of search engine optimization, covered in our SEO guide. Not publishing a link is not enough to hide a page: Google says it is “almost impossible” to keep a site secret that way.

How does a web crawler work?

A web crawler starts from known URLs, downloads each page, extracts its links and puts the new addresses in a queue. There is no central registry: Google discovers URLs through the links on known pages and through sitemaps (guide to how Google Search works, updated on December 18, 2025).

1. Known URLs: pages already crawled, links found elsewhere, URLs from an XML sitemap.
2. robots.txt: the robot reads the site’s rules before crawling its pages.
3. Download: an HTTP request returns a status code and the HTML.
4. Links: the page’s <a href> links join the queue.
5. Rendering: Google runs the JavaScript with a recent version of Chromium and collects the links from the rendered HTML.
6. Return: the robot comes back to known pages, at a pace it chooses.

Googlebot uses an algorithm to decide which sites to crawl, how often and how many pages to fetch from each one. It slows down based on the server’s responses: for Google, HTTP 500 errors mean “slow down”.

Crawling is not indexing

Google describes three stages, crawling, indexing and serving search results, explained in our page on how Google works. Not every page makes it through each stage.

For the method, see how to get a page indexed.

What are the different types of crawlers?

The robots that crawl the web fall into six families, depending on what they do with pages and how they treat robots.txt. At Google, Googlebot always obeys robots.txt for automatic crawling, but special-case crawlers such as AdsBot may depart from it under an agreement with the site.

User-triggered fetching therefore partly escapes robots.txt. At OpenAI, each bot is configured separately: blocking GPTBot does not block OAI-SearchBot (overview of OpenAI’s crawlers, undated). Anthropic states that its bots obey robots.txt and do not bypass CAPTCHAs (Claude Help Center).

To decide which AI bots to let through, see our page on AI crawlers.

What does Googlebot do, according to Google?

Googlebot is the Google Search crawler. The name covers two crawlers, Googlebot Smartphone and Googlebot Desktop, which share the Googlebot token in robots.txt; most requests come from the mobile crawler.

Google lists other common crawlers in its overview of crawlers (June 12, 2026): Googlebot Image, Video and News, Storebot-Google (Shopping), Google-InspectionTool (testing tools) and GoogleOther. The Googlebot Smartphone User-Agent string ends with (compatible; Googlebot/2.1; +http://www.google.com/bot.html).

How do you control the crawlers that visit your site?

The robots.txt file tells robots what they may crawl. RFC 9309 states that these rules “are not a form of access authorization”.

Born from a consensus reached on June 30, 1994 among robot authors and recorded by Martijn Koster, the protocol was standardized by the IETF in September 2022. RFC 9309, signed by Koster and three Google authors, requires crawlers to parse at least 500 KiB of the file and allows a 24 hour cache. Google applies both values. For Google, robots.txt is mainly there to avoid overloading a site, not to remove a page from search results.

Google then raises the rate again on its own (Reduce the Google crawl rate, October 6, 2026). The same page attributes most crawl spikes to facets, filters and sorting, and to calendars with endless URLs.

To find out whether a crawl limit affects your site, see our page on crawl budget. The syntax and the WordPress case are in our robots.txt guide.

How do you recognize a real Googlebot?

The Googlebot User-Agent is often spoofed. Google recommends a two-way DNS lookup: the IP address must resolve to a name in googlebot.com, google.com or googleusercontent.com, and that name must point back to the same IP. On the command line: host <IP address>, then host <resulting name>. Google also publishes its IP ranges in JSON.

Common Crawl also reports fake CCBots; the real one has a reverse DNS in crawl.commoncrawl.org. Bing provides a “Verify Bingbot” tool. These checks apply to the addresses in your server logs: see our log file analysis method.

What is an SEO crawler?

An SEO crawler is software that crawls a site the way a search engine bot would, to record its HTTP status codes, links, click depth and tags.

Its limit: it simulates crawling at a given moment. What Googlebot actually did shows up in the server logs and in the Search Console Crawl Stats report, which Google says a site with fewer than a thousand pages “should not need to use”.

Screaming Frog SEO Spider is free up to 500 URLs; a licence unlocks crawling beyond that and advanced features, such as JavaScript rendering. Elev8 Lab has no commercial relationship with the vendors mentioned.

Example: what a crawler sees on elev8-lab.fr

Read-only crawl run by Elev8 Lab on October 19, 2026 on elev8-lab.fr, the French version of the site: starting from the home page, a Python script follows the domain’s <a href> links, without URL parameters, obeys robots.txt and identifies itself with its own User-Agent.

The robots.txt only closes the WordPress admin and login areas (four rules) and allows search engine and AI bots by default. Hence no blocked URL.

All the French articles and guides are 2 clicks or fewer from the home page; further down, only pagination pages and author archives remain.

The 20 pages outside the sitemaps are these listing pages: the crawler finds them through links, because the sitemap helps discovery without limiting it. No 404 error or redirect showed up.

The pages measured weigh about 8% of Googlebot’s 2 MB limit. The page /seo/budget-crawl/ (the French crawl budget article) is served with gzip, but the limit applies to the uncompressed data: its 160 KB are what count. The download times, measured remotely, are only an order of magnitude.

Crawler or scraper: what is the difference, what are the rules?

A web crawler discovers and moves through pages; a scraper extracts targeted data from them to reuse it.

Google classifies as spam the scraping that republishes other sites’ content without added value or citation, in order to manipulate rankings, as well as unauthorized scraping of its results.

Under EU law, Article 4 of Directive (EU) 2019/790 creates a text and data mining exception for lawfully accessible content. It does not apply if rightholders have reserved this use “in an appropriate manner, such as machine-readable means”. Since August 2, 2025, Article 53 of the AI Act (Regulation (EU) 2024/1689) has required providers of general-purpose AI models to put in place a copyright compliance policy that identifies and complies with this reservation.

For personal data, in France, the CNIL fact sheet on web scraping of June 19, 2025 (the CNIL is the French data protection authority) states that scraping “is not prohibited per se” but must be assessed case by case, with mandatory data minimization. In its view, failing to exclude sites that object through robots.txt or CAPTCHAs goes beyond people’s reasonable expectations. It mentions other signals (ai.txt, TDMRep, noai tags) and notes that no single standard exists.

To run your own web crawler, reuse the rules of our crawl: robots.txt obeyed, an identifiable User-Agent, one request per second. If personal data is involved, add the case-by-case assessment that the CNIL requires in France.