Crawl budget rarely gets attention until something has already gone wrong — new pages taking weeks to get indexed, updated content not being reflected in search results, or a migration that seems to have stalled halfway through. By the time it’s visible as a symptom, the waste has usually been accumulating for a long time.

What crawl budget actually is

Crawl budget is the number of URLs Googlebot is willing and able to crawl on your site within a given period, determined by two things: crawl rate limit (how much your server can handle without degrading) and crawl demand (how much Google actually wants to crawl your site, based on perceived value and freshness). Neither is unlimited, and on large or technically messy sites, the budget gets spent in the wrong places long before it reaches the pages that matter.

Where the budget actually goes

A crawl budget audit starts with server log files, not Search Console — log files show what Googlebot actually requested, not a sampled summary. The patterns that show up consistently:

• Faceted navigation and filter parameters generating thousands of low-value URL combinations that get crawled repeatedly
• Redirect chains where Googlebot burns multiple requests resolving a single destination
• Orphaned or near-duplicate pages from old CMS migrations still being recrawled out of habit
• Session IDs or tracking parameters in URLs creating infinite near-duplicate variations
• Soft 404s — pages returning a 200 status but showing “no results” or empty content, which Google keeps recrawling hoping something changes

On one recent audit, a faceted navigation bug on an e-commerce site was generating over 4,200 duplicate indexed URLs from filter combinations no user would ever intentionally construct — and log file analysis showed Googlebot spending more requests on those combinations than on the site’s actual product pages.

Running the audit

1. Pull raw server logs covering at least 30 days of Googlebot activity — not a Search Console sample.
2. Segment by URL pattern to see where crawl requests concentrate, and cross-reference against which URL patterns actually drive organic traffic or revenue.
3. Compare crawl frequency to page value. Pages that matter should be crawled often. Pages generating no value should barely be crawled at all — if the ratio is inverted, that’s the waste.
4. Check response codes at scale. A high proportion of redirects, errors, or soft 404s in the crawled set is a direct signal of wasted requests.
5. Cross-reference against the XML sitemap and indexed page count in Search Console to see how much of what’s being crawled should even be eligible for indexing.

Fixing it

Once the waste is located, the fix is usually a combination of robots.txt disallow rules for genuinely low-value parameter combinations, canonical consolidation for near-duplicates, cleaning up redirect chains to single-hop, and noindexing thin or soft-404 pages rather than leaving them live. None of this is exotic — the value is entirely in finding where the budget is actually leaking before spending time on anything else.

Sites that fix crawl budget waste typically see faster indexation of new content and fresher recrawls of updated pages within weeks — not because Google suddenly cares more about the site, but because the requests it was already making are finally being spent on pages worth crawling.