By Rahul Saini, Author at Search Counsel Co. Last updated [JULY] 2026.

1) What crawl budget actually is

Crawl budget is the amount of crawling Google is willing to do on your site. Google’s own definition is worth keeping in mind: it’s “the number of URLs Googlebot can and wants to crawl.” Those two verbs, can and wants, are the whole concept. Crawl budget is set by two separate things.

Your effective crawl budget is whichever of the two is lower. If your server can handle more but Google doesn’t want to crawl more, crawling stays flat. If demand is high but your server is slow, Google throttles itself to protect your site. One more detail that trips people up: crawl budget is set per hostname, so your main site and a subdomain each get their own. For how crawling fits with rendering and indexing overall, see the technical SEO pillar.

2) Does crawl budget matter for your site?

For most sites, no. If you have a few thousand pages or fewer, Google crawls the whole site comfortably in a single pass, and crawl budget is simply not your bottleneck. Chasing it on a small site is time stolen from content and links, which matter far more at that scale.

Crawl budget becomes a real concern when one of these is true:

• You run a large site. Tens of thousands of URLs and up, think ecommerce catalogs, marketplaces, directories, or big publishers. As a rough rule of thumb, it’s worth attention at around 10,000 URLs and a genuine priority in the hundreds of thousands.
• You publish or change pages faster than Google indexes them. News and large content sites where fresh pages need to be found quickly.
• You have a lot of “Discovered, currently not indexed” URLs. When Search Console shows many known-but-uncrawled pages, crawl budget is often the cause. That status is covered in our guide to fixing indexing problems.

Worth saying clearly: crawl budget is not a ranking factor. It affects how fast your pages get discovered, indexed, and refreshed, which then feeds into rankings indirectly. Fixing it won’t lift a page that’s already indexed and crawled daily.

3) What wastes crawl budget

Most crawl-budget problems aren’t a shortage of crawling. They’re crawling spent on the wrong URLs. Google has named the usual culprits, and they’re the same ones that show up in almost every large-site audit:

• Faceted navigation. Filter and sort combinations can generate thousands of near-duplicate URLs from a few hundred products.
• Session IDs and tracking parameters that create a new URL for the same page.
• On-site duplicate content, so Google crawls several versions of one thing.
• Soft 404s, pages that return a 200 status but are effectively empty.
• Infinite spaces like endless calendars, internal search results, and “load more” paths that never end.
• Hacked pages and spam, which Google crawls before it can judge them.
• Redirect chains and broken links, where each hop or 404 is a crawl that produces nothing indexable.

The pattern is simple. Every crawl Google spends on a junk URL is a crawl it didn’t spend on a page you care about.

4) How to optimize crawl budget

Optimization is two moves: help Google crawl faster, and stop it wasting crawls on pages that don’t matter. Here are the tactics that move the needle, in rough order of impact.

• 1. Speed up your server. Server response time is the main lever on how fast Googlebot crawls, so this is where the biggest gains are. Caching is the highest-impact change: a cached page can return in tens of milliseconds instead of hundreds or thousands. Faster responses let Google use more connections and crawl more per visit. Our site speed guide covers the how.
• 2. Stop Google crawling junk. Block infinite spaces, faceted-filter combinations, and internal search results in robots.txt. Google specifically recommends robots.txt for these. Note that a noindex tag won’t save crawl budget, because Google still has to crawl the page to see the tag. To stop the crawl, you need robots.txt.
• 3. Kill duplicate content. Consolidate near-duplicate URLs with canonical tags and clean parameter handling, so Google isn’t crawling ten copies of one page.
• 4. Keep your sitemap clean. List only canonical, indexable URLs that return a 200 status, with accurate lastmod dates so Google prioritizes what actually changed. Drop anything redirected, noindexed, or gone.
• 5. Fix broken links and redirect chains. Point redirects straight at a final 200 URL and repair or remove broken internal links. Our guide to finding and fixing crawl errors walks through this.
• 6. Flatten and interlink. A flat architecture with strong internal links means important pages are only a few clicks deep and easy to discover. Orphan pages, with no internal links pointing to them, get missed.
• 7. Ease off client-side rendering. Rendering JavaScript is expensive, so pages built entirely in the browser cost more crawl resources than server-rendered ones. Serving your important content server-side helps Google process the site more efficiently. See our JavaScript SEO guide.

5) How to check your crawl budget

You don’t get a crawl-budget number, but you can watch how Google spends it. The Crawl Stats report in Search Console (Settings, then Crawl stats) is the place to start. It shows total crawl requests over time, average server response time, host status, and a breakdown of what Google crawled by response code, file type, and purpose (discovering new URLs versus refreshing known ones).

A few things to look for. A rising average response time means your server is becoming the bottleneck. A sudden spike in crawl requests can mean a migration introduced redirect chains, or a noindex tag was removed and Google is now crawling pages it used to skip. And a big gap between the URLs in your sitemap and the URLs actually indexed is a sign crawling is being wasted somewhere.

For a deeper view, analyze your server logs. They record every request each crawler actually made and the status it got back, so they show exactly which URLs Googlebot spends its budget on, and how often. Tools like Screaming Frog and Sitebulb, plus dedicated log analyzers, make this manageable on large sites.

6) Crawl budget in the AI era

Crawl budget used to be a story about one crawler. In 2026 it’s about a crowd. Automated bots overtook humans as the majority of web traffic for the first time this year, and AI crawlers are a big part of that surge. By one network’s data, AI and search crawler traffic grew sharply over a single year, with Googlebot up around 96% and OpenAI’s GPTBot up over 300%. Alongside GPTBot sit ClaudeBot, OAI-SearchBot, PerplexityBot, and dozens more, all pulling pages from the same servers.

Here’s why that matters for crawl budget. Google slows its crawling when your server slows down or returns errors. So a wave of AI bots hammering your site can indirectly throttle Googlebot, which means slower indexing of your real content. Crawl-budget management has quietly expanded into AI-bot governance.

The practical version: check your server logs for AI bot user agents. If training bots like GPTBot and ClaudeBot are consuming real resources and you don’t need to be in model training data, you can slow or block them in robots.txt or at your CDN with little cost to search visibility. Be careful with retrieval bots like OAI-SearchBot and PerplexityBot, though, since blocking those removes you from AI answers. That trade-off is exactly what our robots.txt for the AI era guide and the AI search optimization hub are built to help you decide.

7) Crawl budget myths worth dropping

Crawl budget attracts bad advice. Four myths in particular waste people’s time:

• “noindex saves crawl budget.” It doesn’t. Google has to crawl a page to read the noindex tag, so the crawl still happens. Use robots.txt to stop the crawl.
• “crawl-delay controls Googlebot.” It doesn’t. Google ignores the crawl-delay directive entirely. Google also retired the manual crawl-rate limiter in Search Console back in early 2024, and Googlebot now adjusts its own rate based on how your server responds.
• “Blocking pages boosts the rest.” Only if your server was overloaded. Otherwise, freeing up crawls doesn’t automatically redirect them to your good pages.
• “Crawl budget is a ranking factor.” It isn’t. It affects discovery and indexing speed, which influence rankings only indirectly.

8) Sources used for this guide

This guide leans on Google’s own documentation for the mechanics of crawl budget, and on named network data for the AI-crawler figures.

FAQ: crawl budget

Conclusion: spend the budget where it counts

Crawl budget is simple once you strip away the hype. It’s how much Google can and wants to crawl, and for most sites it’s a non-issue. If you run a large or fast-moving site, the job is to make crawling efficient: a fast server, a clean sitemap, no junk URLs, and a flat, well-linked structure. Watch the Crawl Stats report, keep an eye on AI-bot load, and let Google spend its attention on the pages that actually earn you traffic.

From here, the two neighboring pieces of the puzzle are the errors that waste crawls and the reasons a crawled page still might not get indexed. Both are covered in our guides to finding and fixing crawl errors and fixing indexing problems, and both sit under the technical SEO pillar.