Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- Technical SEO
After moving to a new host or CDN, have you seen Googlebot’s crawl rate reset and ramp up slowly? How long until it matched old levels?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
A dip in Googlebot’s crawl rate after moving to a new host or CDN is common, and it usually recovers within days to a few weeks, provided the new infrastructure responds quickly and reliably. Google sets how much it crawls partly by how the server responds: fast, successful responses allow more crawling, while slow responses and errors reduce it. A new host, new IP addresses or a new CDN gives Google a changed environment to reassess, and if the move coincides with configuration problems, such as bot rules, rate limits, cache misses or slower origin responses, the dip is deeper and longer. When crawl does not recover within a few weeks, something about the new set-up is usually the reason, not a waiting period.
What typically happens after a move
Phase Typical observation Cutover and first days Some fluctuation in crawl requests and response times; DNS changes propagate First one to two weeks Crawl rate rises again if responses are fast and error-free After a few weeks Crawl back to previous levels, or higher if the new infrastructure is faster If not recovered Look for errors, blocks or slow responses on the new set-up Common reasons recovery is slow
- Security rules on the new CDN challenging, rate-limiting or blocking Googlebot, sometimes intermittently.
- Slower responses because pages are not cached at the edge for crawlers, or the origin is further away.
- Errors under load that did not appear in testing.
- robots.txt problems on the new configuration, such as a different file or intermittent errors.
- Redirect or canonical changes introduced with the move, which change what Google has to crawl.
What to monitor
- Crawl stats in Search Console: total requests, average response time, response codes and host status, day by day around the move.
- Server and CDN logs: verified Googlebot requests, status codes and response times at the edge and origin.
- Security logs: challenges or blocks applied to Google’s IP ranges.
- Indexing of new content: how quickly new pages are crawled and indexed compared with before.
Preparing for a move
- Configure bot management to allow verified search crawlers before cutover.
- Make sure caching rules serve Googlebot cached pages where appropriate, rather than bypassing the cache.
- Serve robots.txt from the edge and test it on the new set-up.
- Load test the new infrastructure with realistic crawler and user traffic.
- Avoid combining the move with URL changes, redesigns or other migrations.
- Record baseline crawl and response metrics beforehand so you can compare.
Hypothetical example:
A publisher moves from a traditional host to a CDN-fronted set-up. Crawl requests fall sharply in the first week and do not recover in the second. Crawl stats shows average response time up, and logs show Googlebot’s requests bypassing the CDN cache because the site’s cookie-based cache rule treats all requests with certain headers as uncacheable. After the cache rule is fixed, response times fall below the old host’s, and within ten days Googlebot’s daily requests exceed their previous level. The team had assumed it was waiting for Google to “trust” the new host; the problem was a configuration it could fix.
Watch for this:
Intermittent blocks are the hardest to spot. A bot rule that challenges a small percentage of Googlebot’s requests may not show clearly in totals, but it can keep crawl rates suppressed. Check security logs specifically for challenges applied to verified Google IP addresses.
Moves that change IP ranges and geography
Some moves change more than the provider: the site may now be served from different regions, through different edge locations, or from shared IP addresses. For most sites that has no lasting effect on crawling or rankings, since Google crawls from its own infrastructure and handles CDNs routinely. What matters is that responses from the edge to Google’s crawlers are fast and consistent. If the new set-up routes crawler requests to a distant origin because of cache rules, or serves them from a region with higher latency, response times rise and crawl rates follow. Measuring response times for verified crawler requests specifically, rather than for all traffic, shows whether crawlers are getting the same experience as users.
When to worry
A dip of a few days after cutover is normal and not a reason for concern. If crawl requests remain well below baseline after two or three weeks, or new content is taking much longer to be indexed, investigate the new set-up systematically: responses, errors, blocks and caching. Google does not publish a fixed recovery period, and in practice recovery tends to follow the server’s behaviour rather than a timetable.
Expect a short dip, prepare the new infrastructure for crawlers before cutover, monitor Crawl stats and logs closely, and treat a slow recovery as a sign of a fixable problem rather than something to wait out.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.