Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- Technical SEO
How much log data do you need before crawl-frequency conclusions are reliable? Is a two-week sample enough on a site with strong weekly seasonality?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Two weeks is enough for some questions and not for others. It is usually enough to describe total Googlebot activity, status codes and the crawl rate of frequently crawled sections, as long as you compare like weekdays. It is not enough to estimate crawl frequency for the long tail, because many URLs on a large site are crawled less than once in two weeks, and a short window records them as “not crawled” when they are simply crawled rarely. Nor is it enough to measure the effect of a change, where you need several full weekly cycles before and after. On a site with strong weekly seasonality, I would want at least 60 to 90 days of retained logs for any conclusion about crawl frequency by URL.
Match the sample to the question
Question Minimum useful window Why How much does Googlebot crawl, and what status codes does it get? Two to four full weeks High-volume totals stabilise quickly if weekly cycles are handled How often are important, frequently crawled pages fetched? Four weeks Several fetches per URL give a stable interval How often is the long tail crawled? 60 to 90 days or more Many URLs are fetched monthly or less; short windows cannot see them Did a change affect crawling? At least four full weeks before and four after, ideally with a control segment Weekly seasonality and normal drift need to be separated from the change Seasonal patterns across the year A year Crawl can follow the site’s own seasonal publishing and demand The censoring problem
A two-week sample creates a bias that is easy to miss. URLs crawled every three weeks might appear once or not at all. If you count “URLs not crawled in the window” as uncrawled, you overstate the problem; if you calculate average intervals only from URLs seen twice, you understate how long intervals really are, because the slow URLs drop out of the calculation. Statisticians call this censoring. The fix is a longer window, or methods designed for it, such as time-to-recrawl analysis that treats “not yet crawled again” as an incomplete observation rather than a missing one.
Handling weekly seasonality
- Use whole weeks. A window that starts on a Monday and ends on a Wednesday overweights some days.
- Compare like days. Compare Mondays with Mondays, or use seven-day rolling totals.
- Watch your own publishing rhythm. If you publish heavily on certain days, Googlebot often follows, because new URLs and updated sitemaps attract crawling.
- Separate server-side causes. Maintenance windows, deployment days and traffic peaks can change response times, and Google adjusts crawl rate when a server slows or errors.
Getting clean data
- Verify Googlebot. Filter requests by Google’s published crawler IP ranges or by reverse DNS lookup. Spoofed user agents are common and distort any analysis.
- Separate Google’s crawlers. Googlebot smartphone, desktop, image and other Google fetchers behave differently. User-triggered fetchers are not crawling at all.
- Include CDN logs. If a CDN serves cached responses, origin logs miss those requests.
- Normalise URLs. Group parameters, trailing slashes and case variants, or crawl frequency per page will be split across duplicates.
- Keep logs long enough. Many teams rotate logs after a week or two. Set up retention of at least 90 days for Googlebot requests, even if only in summarised form.
Expressing uncertainty
For headline figures such as crawl requests per day or the share of URLs crawled in a period, bootstrap by day: resample days from the window many times and compute the figure each time. The spread shows how much the result depends on which days you happened to capture. If the spread is wide, you need more data before making a decision. For crawl intervals by segment, report medians with the share of URLs not seen in the window, so readers understand how much of the segment the estimate actually covers.
Hypothetical example:
A classifieds site analyses two weeks of logs and concludes that 60% of its listing pages are never crawled. Extending the window to 90 days shows that most of those pages are crawled every three to six weeks, and only about 15% are genuinely not crawled in the period. The two-week figure would have led the team to prune pages that Google was, in fact, visiting. The 90-day analysis instead points to a smaller set of orphaned listings as the real problem.
A standing setup beats one-off samples
The best answer to “how much log data do I need” is to stop sampling and keep a continuous, summarised record: verified Googlebot requests per URL per day, status code, response time and bytes. That table is small compared with raw logs, can be kept for a year or more, and lets you answer any of the questions above with the window it needs. Search Console’s Crawl stats report is a useful companion for totals and host status, but it is sampled and does not give URL-level frequency.
Two to four weeks for totals and status codes, 60 to 90 days for crawl frequency across the long tail, whole weeks and like-for-like days always, and a continuous summarised log store so the next question does not start with a two-week sample.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.