Discussion
  • Crawling
  • Technical SEO

Google’s crawl budget guide targets 1M+ page sites, or 10k+ pages changing daily. Seen crawl limits bite on smaller sites? What was the real cause?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Yes, crawl limits can bite well below those thresholds, but on smaller sites the cause is rarely that Google has handed out too little crawl. It is usually one of four things: the server or a security layer is telling Googlebot to back off, the site exposes far more crawlable URLs than it has real pages, Google has little interest in the URLs it has found, or crawling is fine and the real problem is indexing. The thresholds in Google’s guide describe when active crawl budget management is worth the effort. They don’t mean a 5,000-page site is immune to the mechanics behind it.

    The practical test is simple to state. If Googlebot’s request volume drops when your server slows down or errors, you have a capacity problem. If requests are healthy but go to the wrong URLs, you have a URL-space problem. If the server is fast, the URL space is clean and new pages still sit unvisited, you have a demand problem, and no crawl setting will fix it.

    What Google’s guide actually says about who needs it

    Google’s crawl budget guide is written for large sites (around 1 million or more unique pages changing roughly weekly), medium sites (10,000 or more pages changing daily), and, often overlooked, sites with a large share of URLs in “Discovered - currently not indexed”. That third group has no size limit. A 3,000-page site where half the URLs sit in that status is inside the guide’s audience.

    Gary Illyes has also played down page count as the main variable. On Google’s Search Off the Record podcast he said “I would say 1 million is okay probably”, and pointed to server cost instead: “If you are making expensive database calls, that’s going to cost the server a lot”. That matches what shows up on small sites with problems: the page count is modest, but each request is slow or expensive to serve.

    Crawl budget is two numbers, not one

    An analogy helps here. Think of Googlebot as a courier on a daily route. Capacity is how many parcels your loading dock can take before the courier backs off. Demand is how many parcels the depot thinks you actually want. A small shop with a jammed dock gets fewer deliveries regardless of its size. A small shop that keeps ordering junk uses its deliveries on the wrong parcels.

    Technically, Google’s guide describes a crawl capacity limit, which rises when the site responds quickly and falls when responses slow down or return server errors, and crawl demand, driven by perceived inventory, popularity and staleness. The guide also says every URL Googlebot fetches counts, including CSS, JavaScript and XHR requests made during rendering, not just HTML pages. There has been no Search Console setting to raise the crawl rate since Google retired the crawl rate limiter tool in early 2024, so the only levers are server health and what you ask Google to crawl.

    The real causes on sites below the thresholds

    Cause How it shows up Where to confirm it Usual fix
    Slow or expensive responses Crawl requests fall as average response time rises; uncached, database-heavy templates Crawl Stats average response time; server logs filtered to verified Googlebot Page caching, query optimisation, better hosting tier
    Security layer throttling Googlebot 429 or 5xx responses to Googlebot only; challenges from bot protection or rate limiting Crawl Stats “By response”; WAF and CDN logs Allow verified Googlebot, loosen rate limits for it
    robots.txt fetch failures Crawling pauses sitewide for no obvious reason Crawl Stats host status: robots.txt fetch Serve robots.txt reliably with a 200 or 404, never a 5xx or timeout
    URL inflation Most Googlebot hits go to filters, sort orders, internal search, calendars or tracking parameters Logs: unique URLs crawled versus canonical pages Stop generating crawlable links to them; disallow parameter patterns you never want crawled
    Rendering overhead A high share of “Page resource load” requests; many API calls per page Crawl Stats “By Googlebot type” and “By file type” Server rendering or fewer client-side fetches; stable resource URLs without cache-busting parameters
    Low crawl demand Fast server, clean URLs, but new pages wait weeks; growing “Discovered - currently not indexed” Page indexing report; time to first Googlebot hit in logs Better internal links, fewer weak pages, more reasons for others to link

    The robots.txt row deserves a note because the effect is sitewide. Google’s robots.txt documentation treats 5xx and 429 responses on the robots.txt file as a temporary full disallow. A misconfigured firewall rule that challenges requests for /robots.txt can pause crawling on a 200-page site as effectively as on a 2 million page one.

    URL inflation is why page count is a poor guide. A store with 4,000 products and six filter types can generate hundreds of thousands of crawlable combinations. That site is “small” in the merchandising sense and large in the only sense Googlebot sees.

    Hypothetical example:

    A 6,000-product WooCommerce store on shared hosting adds layered navigation. Server logs show most verified Googlebot requests going to ?filter_colour= and ?orderby= URLs, each taking over a second to generate because the filter queries are uncached. The host starts returning intermittent 503s under load. Search Console shows Googlebot requests falling and new products taking weeks to be crawled. Neither the page count nor Google’s allocation is the problem. The fix is to stop linking crawlable filter combinations, cache or speed up the remaining templates, and let crawl capacity recover.

    Where Google’s explanation and field observations differ

    Google’s documented guidance for “Discovered - currently not indexed” says the page was found but not crawled yet, typically because Google expected crawling it to overload the site and rescheduled it. That is a capacity explanation.

    The practical complication is that many small sites show large volumes in this status while their servers are fast and error-free. Practitioners commonly find these URLs are thin, near-duplicate or auto-generated, and the status clears after consolidation or stronger internal linking rather than after server work.

    Those observations don’t necessarily contradict the guidance. Google’s scheduling reflects both capacity and demand, and the report shows one status for both. So treat “Discovered - currently not indexed” as a signal to investigate, not proof of a capacity problem. If Crawl Stats shows healthy response times and no host issues, low demand is the more likely reading.

    Watch for this:

    “Crawled - currently not indexed” is not a crawl budget problem. Google fetched the page and decided not to index it. Blocking other URLs to “free up budget” won’t change that decision. It is a quality, duplication or canonical question.

    One newer source of confusion: Googlebot fetches only the first 2MB of an HTML file, and anything beyond that is not fetched, rendered or indexed. A page builder that inlines huge JSON blobs or SVGs can push links and content past that point. Pages linked only from the truncated part then look like a discovery failure when the real issue is page weight.

    The diagnostic sequence I would follow

    Crawl Stats: are requests falling while response time or errors rise?
    ↓
    Yes: capacity problem. Check host status, 5xx and 429 by date, WAF rules, hosting limits
    ↓
    No: pull 30 days of logs for verified Googlebot. What share of hits reach canonical, indexable URLs?
    ↓
    Low share: URL-space problem. Find the patterns generating crawlable junk
    ↓
    High share, but new pages still wait: demand problem. Work on page value and internal links
    1. Crawl Stats report. Look at total requests, average response time, host status (robots.txt fetch, DNS, server connectivity), and breakdowns by response, file type, purpose (discovery versus refresh) and Googlebot type. Other Google crawlers such as AdsBot and StoreBot appear here too, which matters on stores running Shopping or ads.
    2. Server logs. Verify Googlebot by reverse DNS or Google’s published IP range files, which moved in March 2026. Measure unique URLs crawled, status codes and response times for Googlebot specifically. A WAF can serve Googlebot differently from your browser.
    3. Page indexing report. Compare “Discovered - currently not indexed” with “Crawled - currently not indexed”, and open example URLs from each. The pattern in the examples usually tells you whether it is a template, a parameter or a content type.
    4. Time to first crawl. For pages published in the last 60 days, measure the gap between publication and the first verified Googlebot hit. This is the most direct measure of whether crawling is actually limiting you.

    How to confirm the fix worked

    Change one cause at a time where you can, and add a Search Console annotation on the release date. Track three numbers for four to six weeks: Googlebot requests to canonical URLs, average response time for Googlebot, and median time to first crawl for new pages. Crawl activity also rises around migrations and large content changes, so don’t read a spike in the week after a release as proof on its own.

    What would change the diagnosis: if Googlebot requests climb after you remove 5xx errors or speed up templates, capacity was the constraint. If requests stay flat but shift to canonical URLs after you close off parameters, URL space was the constraint. If the server is fast, the URL space is clean and new pages still sit unvisited, stop tuning crawl settings. At that point the site needs pages that Google and other sites consider worth fetching, and that is a content and linking problem rather than a crawling one.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.