Crawl Budget Forensics with Log Files: What Googlebot Actually Wastes Time On

GSC's Crawl Stats report tells you Googlebot hit your site 1.2 million times last month. Useful. It does not tell you that 340k of those hits were on a faceted filter nobody should be indexing, or that your 301 chain to the new product URLs is burning a quarter of your budget. For that, you need the raw logs.

This is the pipeline we run for clients with 100k+ URLs, and the recurring patterns we find once the data is in a queryable shape.

Why GSC's Crawl Stats isn't enough

The Crawl Stats report is sampled, aggregated, and bucketed into categories Google chose for you. You get response code breakdowns, file type splits, and a handful of example URLs per bucket. On a site with a few thousand pages that's fine. On a programmatic site it hides exactly the information you need: which URL patterns are being over-crawled, which templates Googlebot is quietly ignoring, and where redirect chains or soft 404s are eating budget.

Raw logs give you every hit, every timestamp, every response code, every byte. Once you can group by URL pattern and user agent, the waste is usually obvious within an hour.

What counts as "crawl budget" in 2026

Google has been consistent that crawl budget is a function of two things: how much your server can handle (crawl capacity) and how much Google wants to crawl you (crawl demand). You influence capacity with response times and error rates. You influence demand with freshness signals, internal linking, and — crucially — by not wasting the budget you already get.

The forensics work is almost entirely about the second half. You cannot make Google want to crawl you more by cleaning up waste, but you can redirect the budget it already allocates toward URLs that matter.

The pipeline

We keep this intentionally boring. Logs land in object storage, get parsed into a columnar format, and end up in a warehouse where SEO and engineering can both query them.

Collecting logs

From a CDN (Cloudflare, Fastly, CloudFront) you want the full access log stream, not sampled analytics. From an origin behind the CDN you want logs too, because CDN cache hits never reach origin and you'll miss cache misses vs hits otherwise.

Minimum fields:

• Timestamp (with timezone)
• Client IP
• Request method and full path (including query string)
• Response status
• Response bytes
• User-Agent string
• Referer
• Cache status (hit/miss/bypass) if CDN
• Response time

Parsing and verifying Googlebot

User-Agent strings are trivial to spoof. Before you trust a hit as Googlebot, verify it with a reverse DNS lookup on the IP, then a forward lookup to confirm the hostname resolves back. Google publishes the IP ranges in JSON now, so you can do a straight range check instead. Either way, do it once at ingest, add a boolean column, and never trust the raw UA again.

import ipaddress
import json
import requests

# Cached daily
GOOGLEBOT_RANGES = requests.get(
    "https://developers.google.com/search/apis/ipranges/googlebot.json"
).json()["prefixes"]

NETWORKS = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"])
            for p in GOOGLEBOT_RANGES]

def is_verified_googlebot(ip: str) -> bool:
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in NETWORKS)

Grouping by URL pattern

This is the step most teams skip, and it's the one that makes the data useful. A URL like /shoes/nike/air-max?color=red&size=42 is one of hundreds of thousands. On its own it tells you nothing. Grouped as /shoes/{brand}/{model}?color={c}&size={s} it tells you a story.

We maintain a small ordered list of regex-to-pattern rules per site, applied at query time as a CASE WHEN, or materialised as a column during ETL. Start with 20–30 patterns covering your main templates, your faceted filters, your pagination, your search pages, static assets, and a catch-all "other" bucket.

The waste patterns we find every time

After running this on dozens of sites, the same handful of problems show up. If you don't have logs yet, assume at least two of these apply to you.

Faceted URLs nobody blocked

The classic. An e-commerce or listing site exposes filters as query parameters, those parameters get linked from the UI, and Googlebot crawls every combination it can find. We regularly see 30–60% of crawl budget on sites with no faceted navigation strategy going to parameter combinations that shouldn't be indexed and often aren't even reachable through intentional navigation anymore.

The fix is boring: decide which facet combinations deserve indexable pages, make those canonical and link to them properly, and block the rest. robots.txt disallow for parameters you never want crawled, canonical tags for parameters that collapse to a parent, and rel=nofollow on internal links that generate noise. Don't rely on noindex alone — the page still gets crawled to see the tag.

Redirect chains after migrations

Every site migration leaves scars. Googlebot hits URL A, gets a 301 to B, which 301s to C, which 301s to D. Each hop is a separate request against your budget. We've seen sites where 15% of Googlebot requests were resolving multi-hop chains left over from migrations two and three years prior.

Query your logs for 301 and 302 responses, group by target, and flatten any chain longer than one hop by updating the redirect rules to point straight to the final destination. Then update your internal links to point at the final URLs so Googlebot stops discovering the old ones.

Soft 404s returning 200

Your "no results found" search page returns a 200. Your out-of-stock product page returns a 200. Your expired job listing returns a 200. Googlebot crawls all of them, repeatedly, because nothing tells it to stop.

Either return a real 404/410 for genuinely dead content, or make the page substantive enough to deserve a 200. The log signal to look for is high crawl frequency on URL patterns that produce low-value HTML — you can cross-reference with GSC's "Soft 404" exclusion report to confirm.

Infinite calendar and ID spaces

A calendar widget with next/previous month links generates infinite URLs. A date-filter parameter that accepts any ISO date does the same. An ID parameter that increments past your actual content range produces thin or empty pages forever. Googlebot will happily walk all of them.

SELECT
  url_pattern,
  COUNT(*) AS hits,
  COUNT(DISTINCT url) AS unique_urls,
  AVG(response_bytes) AS avg_bytes
FROM googlebot_logs
WHERE date >= CURRENT_DATE - 30
GROUP BY url_pattern
ORDER BY hits DESC
LIMIT 50;

Sort by unique_urls descending and anything with hundreds of thousands of unique URLs in a single pattern deserves a hard look.

Static assets eating the HTML budget

Google separates crawl budget somewhat by resource type, but on sites with aggressive cache-busting (every JS/CSS file gets a new hash on every deploy) we still see Googlebot pulling assets repeatedly. Long cache lifetimes with proper Cache-Control headers, and consistent filenames where possible, cut this down. Check your logs for 200s on .js and .css with Googlebot as the UA and you'll see whether this is you.

Turning the data into a weekly loop

One-off audits are useful, but the wins compound when this becomes a weekly report. Our standard dashboard has four charts:

• Verified Googlebot hits per day, split by response code
• Top 20 URL patterns by crawl volume, week over week
• Redirect hop distribution (how many requests resolved in 1, 2, 3+ hops)
• Crawl volume on URL patterns tagged as "should not be crawled"

That last one is the north star. When a new facet, filter, or template ships, it shows up there within days. Engineering gets a ticket, the pattern gets blocked or canonicalised, and the line goes back down.

Where we'd start

If you've never looked at your logs, don't try to build the full warehouse pipeline on week one. Pull one week of CDN logs into DuckDB or BigQuery, verify Googlebot by IP, group by the ten URL patterns you can describe off the top of your head, and sort by request count. You will find something ugly within an hour. Fix that one thing, then come back and build the pipeline properly once you've proven the value internally.

If you want a hand setting this up against your stack, we do this work as part of our technical SEO engagements.