Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- Site Architecture
- Technical SEO
On an enterprise domain with several busy subdomains on shared infrastructure, have you seen a heavy UGC subdomain reduce crawling of the commercial one?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
It can, indirectly. Google manages crawling largely per host, and Search Console reports crawl statistics by hostname, so a UGC subdomain does not simply use up the commercial subdomain’s crawl allowance. But crawl capacity depends on how quickly and reliably the server responds. When several busy subdomains share the same servers, heavy crawling of one can slow responses for all of them, and slower responses lead Google to crawl less, including on the commercial host. The effect shows up as rising response times and falling crawl requests on the commercial subdomain during periods of heavy UGC crawling. If you see that pattern, the fix is to separate the load, reduce unnecessary crawling of the UGC host, and make the commercial host fast regardless of what else is happening.
How to tell whether it is happening
- Compare Crawl stats by host. Look at requests and average response time for the UGC and commercial subdomains over the same period.
- Look for correlation. Do spikes in crawling of the UGC host coincide with rising response times and falling requests on the commercial host?
- Check server metrics. CPU, database load and response times on the shared infrastructure during crawl spikes.
- Check logs for status codes and response times by host for verified Googlebot requests.
- Check indexing speed for new commercial pages during those periods.
Why UGC hosts attract heavy crawling
- Large numbers of URLs: threads, profiles, tags, paginated lists.
- Frequent changes as users post, which invites recrawling.
- Parameter and sorting variants that multiply URLs.
- Spam or low-value pages that should not be crawled at all.
Fixes
Fix Effect Serve the commercial host from separate infrastructure or strongly cached at the edge Its responses stay fast whatever happens elsewhere Reduce crawlable low-value URLs on the UGC host Less load from crawling pages that should not be indexed Noindex thin UGC and block parameter patterns in robots.txt Crawl concentrated on valuable UGC Scale shared infrastructure for peak crawl load Fewer slowdowns overall Monitor response time by host with alerts Early warning when crawling starts to strain servers Hypothetical example:
A software company runs its marketing site, documentation and community forum as subdomains on the same application cluster. Crawl stats show the forum receiving far more Googlebot requests than the marketing site, and on days of heavy forum crawling, the marketing site’s average response time rises and its crawl requests fall. The company moves the marketing site behind edge caching, noindexes forum threads without replies, blocks sorting parameters on the forum, and adds capacity to the cluster. Forum crawling falls to a steadier level, the marketing site’s response times stay flat through crawl peaks, and new marketing pages are indexed faster.
Watch for this:
Rate-limiting Googlebot on the UGC host to protect the servers can reduce indexing of valuable community content and, if applied carelessly at the infrastructure level, affect other hosts too. Reducing the number of crawlable low-value URLs is a better lever than throttling crawlers.
Should you worry if your commercial host is small?
Google’s crawl budget guidance is aimed mainly at very large sites and fast-changing sites. A commercial subdomain with a few thousand pages is likely to be crawled adequately even if crawling slows somewhat. The problem becomes material when the commercial host is large, changes frequently, or depends on fast indexing of new pages, such as product launches or price changes. Prioritise fixes accordingly.
Separate properties make the picture clearer
Verify each busy subdomain as its own property in Search Console, alongside a domain property for the whole site. Per-host properties make it easier to see crawl stats, indexing and performance for each subdomain without mixing them, and to give the right teams access to the right data. The domain property still shows the overall picture. With both, it becomes straightforward to see whether a change in one subdomain’s crawling coincides with changes in another’s, which is the evidence needed to decide whether infrastructure separation is worth the cost.
Ongoing monitoring
Keep a dashboard of crawl requests and response times per host, alongside server load, and review it after large community events, spam waves or UGC growth. Those are the times when a busy subdomain is most likely to affect its neighbours, and early detection keeps the fix simple.
Check per-host crawl stats and response times for correlation, protect the commercial host with separate infrastructure or edge caching, cut low-value crawlable URLs on the UGC host, and monitor response times by host so shared-infrastructure strain is caught early.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.