SEO log file analysis: method and tools

What is SEO log file analysis?

A log file, or access log, records every request received by the server: human visitors, robots, pages, images, scripts. SEO log file analysis filters this log down to search engine robots. It observes their actual path through the site.

A crawler like Screaming Frog simulates a robot. Search Console summarizes what Google is willing to show. Logs, on the other hand, record what happened, request by request. They are the most factual building block of a technical SEO audit.

When is log file analysis useful?

The exercise takes time and requires access to the server. It is mainly justified in three cases:

• A site with more than 1,000 pages, or an e-commerce site whose URLs change often. Below that, Search Console is usually enough. This is a practitioners’ benchmark, not a Google rule.
• Important pages that struggle to get indexed, or many URLs marked “Discovered – currently not indexed”.
• A redesign or a migration, to check that Googlebot drops the old URLs and discovers the new ones.

Google’s guide to crawl budget (updated on July 22, 2026) targets two profiles. Sites with about 1 million pages updated every week. Sites with about 10,000 pages whose content changes every day. Google states that these figures are rough estimates, not thresholds. Our article on crawl budget details these notions.

What does a log file contain?

The most common format is Apache’s “Combined Log Format”, also used by Nginx. The Apache documentation gives this example:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"

Field	Example	SEO use
IP address	127.0.0.1	Check that the robot is genuine
Timestamp	[10/Oct/2000:13:55:36 -0700]	Frequency and regularity of visits
Request	GET /apache_pb.gif	URL crawled, parameters included
HTTP code	200	What the robot received: 200, 301, 404, 5xx
Size	2326	Bytes transferred, heavy resources
Referer	http://www.example.com/start.html	Page the request came from
User-agent	Mozilla/4.08…	Declared robot: Googlebot Smartphone, Bingbot, GPTBot

Some servers add the response time, which is very useful for spotting slow sections. On Apache, logs are often found in /var/log/. With a shared hosting provider, you download them from the customer area.

How do you analyze logs, step by step?

1. Collect the logs from all servers over a representative period, ideally 30 days or more.
2. Normalize formats and URLs: protocol, case, trailing slash, parameters, so you can match them with a crawl.
3. Filter the relevant robots: Googlebot first, then Bingbot, then AI search engine robots if the subject concerns you.
4. Verify that the robots are genuine (see the next section). Fake Googlebots skew every measurement.
5. Separate pages and static resources: images and scripts also consume requests, to be measured separately.
6. Cross-reference with a full crawl and Search Console data, then segment by page type.

As for tools, Screaming Frog Log File Analyser reads Apache and W3C formats (IIS, Nginx, AWS ELB). It verifies robots automatically. GoAccess, which is free, produces reports from the command line. Oncrawl and Botify offer full platforms, priced on quotation. The Screaming Frog guide explains how to prepare the crawl to cross-reference.

How do you check that Googlebot is genuine?

Any script can present itself as Googlebot. Google documents two methods for verifying its robots (page updated on March 20, 2026):

1. Reverse then forward DNS: the host command on the IP must return a name in googlebot.com, google.com or googleusercontent.com. A forward lookup on that name must return the same IP.
2. IP ranges: compare the IP with the JSON files published by Google (common crawlers, special-case crawlers, user-triggered fetchers).

These files moved on March 31, 2026, according to the Search Central blog. Google moved them to developers.google.com/crawling/ipranges/. It says the old locations will be redirected within six months. Update your verification scripts.

The list of Google’s common crawlers lets you identify each user-agent (Googlebot Smartphone, Googlebot Image, etc.). OpenAI likewise publishes its robots (GPTBot, OAI-SearchBot, ChatGPT-User) and their IP ranges on its dedicated page. How to manage them is covered in our article on AI search engine robots.

Which indicators should you track in the logs?

Indicator	How to read it	Typical action
Distinct URLs crawled	Scope actually crawled	Compare with the number of useful pages
Active / crawled pages ratio	Share of the crawl that reaches pages receiving organic traffic	Reduce URLs with no value
Crawl frequency by segment	Neglected or over-crawled sections	Internal linking, depth, performance
HTTP codes of robot hits	Each 301 or 404 is a wasted request	Fix internal links and redirects
Share of Googlebot Smartphone	Consistency with mobile-first indexing	Check mobile parity
Response time by segment	Slow sections are crawled less	Server performance

The most telling comparison sets logs against the crawl. A page found in the crawl but missing from the logs is too deep or poorly linked. A page found in the logs but missing from the crawl is often an orphan. It can also be a remnant of an old version or a junk URL.

What should you do with the results?

The findings translate into actions on the site, rarely on the server alone:

• move rarely crawled strategic pages higher in the site structure and internal linking;
• fix internal links that point to 301s or 404s;
• deal with junk URLs (parameters, filters) that soak up the crawl, then monitor indexing in Google;
• reintegrate or redirect orphan pages that still receive robot visits;
• speed up slow templates, then measure again a month later.

A log file analysis never promises a traffic gain. It establishes what robots do, and therefore the order of fixes. It is part of the broader technical SEO workstream.