Crawl budget and index bloat: keeping search engines focused on pages that matter

Search engines do not crawl every URL on your site every day. They decide how much to crawl based on how fast your server responds and how valuable your pages seem. That allowance is your crawl budget. When it is spent on duplicate, thin, or useless URLs, your important pages get crawled and refreshed less often.

Does crawl budget matter for your site?

For most small sites, no. If you have a few thousand pages and new content gets indexed within days, crawl budget is not your bottleneck.

It starts to matter when:

• You have tens of thousands of URLs or more (ecommerce, marketplaces, listings, publishers).
• Your site generates URLs automatically through filters, sorting, search, or session parameters.
• New or updated pages take weeks to show up in search.
• Search Console shows a large number of "Discovered – currently not indexed" pages.

What index bloat looks like

Index bloat is when far more URLs are crawled or indexed than you would ever want to rank. Common sources:

• Faceted navigation. Every combination of color, size, price, and sort order creates a new URL.
• Internal search result pages that are crawlable.
• Tracking and session parameters such as ?utm_source= or ?sessionid=.
• Thin archive pages: tag pages with one post, author pages, date archives.
• Duplicate paths: http and https, www and non-www, trailing slash variants, uppercase variants.
• Old, expired, or out-of-stock pages that return 200 with nearly no content.

A five-step cleanup process

1. Inventory what search engines see

Compare three lists: URLs in your sitemap, URLs found by a full crawl, and URLs reported in Search Console's Pages report. The gaps tell you a lot. URLs that are crawled but not in your sitemap are often the bloat.

2. Check your server logs

Logs show exactly where Googlebot and Bingbot spend their time. Group requests by URL pattern. If a large share of bot hits go to parameter URLs or filters, you have found your leak.

3. Decide what each URL group should do

For every pattern, pick one outcome:

• Duplicate of another page: Best action: Canonical to the main version, or 301 redirect
• Useful to users, no search value: Best action: noindex, keep links followable
• No value to users or search: Best action: Block in robots.txt or stop generating the links
• Permanently gone: Best action: Return 404 or 410
• Valuable but thin: Best action: Improve and consolidate

One caution: robots.txt stops crawling, not indexing. If a URL is already indexed and you want it gone, use noindex first, let it drop out, and only then consider blocking it.

4. Fix the internal links that create bloat

Crawlers follow links. If every product card links to five filter combinations, you are inviting the crawl. Use buttons or forms for filters that should not be crawled, and make sure your internal links point to canonical URLs.

5. Keep the sitemap clean

Your XML sitemap should list only canonical, indexable pages that return 200. Remove redirects, noindexed pages, and errors. Use accurate lastmod dates so search engines know what actually changed.

Make every crawl cheaper

Crawl capacity also depends on speed. A faster server lets search engines crawl more without straining your site:

• Keep server response times low, especially for HTML.
• Avoid long redirect chains.
• Return proper 304 Not Modified responses when content has not changed.
• Fix server errors quickly. A spike in 5xx responses causes crawlers to slow down.

How to know it worked

After a cleanup you should see, over several weeks:

• Fewer "Crawled – currently not indexed" and duplicate URLs in Search Console.
• A larger share of crawl requests going to important page types in your logs.
• New and updated pages getting indexed faster.

Tie the change to outcomes with before-and-after measurement, so you can show the cleanup was worth the effort.

SearchVoro crawls your site the way search engines do, groups URLs by pattern, flags bloat sources like parameters and thin archives, and tracks indexing over time so you can see when your important pages start getting the attention they deserve.