A server log file is the web server’s own record of every request it receives. Log file analysis reads that record to see exactly which pages search engine crawlers request, how often, and what the server sends back. It is the only source that shows what a crawler did, as opposed to what a tool estimates it did. Hartzer Consulting provides log file analysis as a focused engagement and as one part of a full technical SEO audit.
What Log Files Show That Other Tools Cannot
A site crawler such as Screaming Frog shows what could be crawled. Google Search Console shows a sample of what Google reports. Log files show what happened: every URL Googlebot requested, the date and time, and the response code it received. That makes them the definitive record for questions about crawling.
Questions Log File Analysis Answers
- Which pages does Googlebot crawl most often, and are they the pages that matter?
- Which important pages has Googlebot not requested in weeks or months?
- How much crawl activity is spent on parameter URLs, redirects, and error pages?
- Is Googlebot still requesting URLs from an old version of the site?
- What response codes are crawlers receiving, and do they differ from what a visitor receives?
- Did crawl activity change before or after a drop in rankings?
- Which AI crawlers, such as GPTBot, ClaudeBot, and PerplexityBot, are requesting pages, and which pages?
When Log File Analysis Is Worth Doing
Log file analysis is most valuable on large websites, where a search engine will not crawl every URL and the choice of what it crawls affects what ranks. It is also valuable during and after a website migration, when the logs show whether crawlers are following the redirects, and whenever a site has an indexation problem that crawl data and Search Console do not explain. On a small site with a few hundred pages and no symptoms, it is rarely necessary, and Hartzer Consulting will say so.
What Hartzer Consulting Needs
- Raw access logs covering at least 30 days, and longer where a specific event is being investigated
- Logs from every server and layer that handles requests, including the content delivery network (CDN) if one is used, because requests answered by the CDN never reach the origin server
- The standard fields: date and time, requested URL, response code, user agent, and IP address
Logs in Apache, Nginx, and IIS formats can be used, along with exports from CDN and hosting providers. Log files contain visitor IP addresses. The analysis requires only the requests made by crawlers, so logs can be filtered to crawler traffic before they are supplied.
How the Analysis Is Done
The first step is verification. Many requests that claim to be Googlebot are not. Bill Hartzer confirms genuine search engine crawlers by checking each IP address against the ranges the search engines publish and by reverse DNS (Domain Name System) lookup. The verified requests are then joined to a crawl of the site and to the XML sitemaps. That comparison separates the URLs into groups: pages that are crawled and should be, pages that should be crawled and are not, and URLs that are crawled and should not exist.
Common Findings
- A large share of crawl activity spent on filtered, sorted, or paginated URLs
- Orphaned pages that crawlers still request because old links or sitemaps point to them
- Redirect chains that crawlers follow thousands of times a month
- Server errors that occur only under crawler load and never appear in manual tests
- New sections of the site that crawlers have barely visited
- Pages served differently to crawlers than to visitors
What the Client Receives
- A report on crawl activity by section, by template, and by response code
- Lists of wasted crawl URLs and of important pages that are not being crawled
- Recommended changes to robots.txt, internal links, sitemaps, and redirects
- A summary of AI crawler activity on the site
To request log file analysis, contact Hartzer Consulting or call +1 214 236 4378.