Discussion
  • Crawling
  • SEO Analytics

Crawl frequency from logs as an SEO KPI: a useful leading indicator, or vanity?

Published 1 reply

Discussion starter: an SEO measurement question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Crawl frequency is a useful leading indicator in specific situations and a vanity metric as a general KPI. More crawling is not better in itself: crawl volume can rise because Google is fetching duplicate URLs, parameter combinations or error pages, which is a problem, not a success. What makes crawl data valuable is measuring it against a goal: whether the pages you care about are crawled often enough, whether new pages are crawled soon after publication, and how much crawling is spent on URLs you do not want indexed. Framed that way, crawl metrics give early warning of indexing problems before they show up in rankings.

    When crawl data works as a leading indicator

    • New sections and launches. If Googlebot is not fetching new URLs within days, indexing will not follow.
    • Migrations. Crawling of redirects and new URLs shows how quickly Google is processing the move.
    • Large sites with more URLs than Google crawls. A falling share of priority pages crawled each month often comes before falling indexing.
    • Fresh content. News, prices and inventory need recrawling quickly; crawl delay translates directly into outdated results.
    • Technical incidents. Sudden drops in crawl or spikes in errors flag server, CDN or robots.txt problems early.

    Better metrics than total crawl volume

    Metric What it tells you
    Share of priority URLs crawled in the last 30 days Whether important pages are being refreshed
    Median time from publication to first crawl for new URLs How quickly new content is discovered
    Share of crawl spent on non-indexable URLs Crawl waste: parameters, redirects, errors, noindexed pages
    Share of crawl returning errors Server and routing health as Googlebot experiences it
    Time from publication to indexing Whether crawling is turning into indexing

    These metrics have a direction that is clearly good or bad, which is what a KPI needs. Total requests per day does not.

    How to calculate them

    1. Keep verified Googlebot requests from edge or server logs, joined to a URL inventory that marks each URL as priority, indexable, non-indexable or unknown.
    2. Calculate the metrics weekly or monthly, by section or template.
    3. Add publication dates from the CMS to measure time to first crawl.
    4. Use URL Inspection samples or Search Console data to connect crawling with indexing.

    Hypothetical example:

    A marketplace reports “Googlebot requests per day” to leadership and celebrates when it doubles. Log analysis later shows the increase came from a new filter module generating crawlable parameter URLs; the share of crawl spent on non-indexable URLs had risen sharply, and new listings were taking longer to be crawled. The team replaces the metric with three: share of new listings crawled within 48 hours, share of crawl on non-indexable URLs, and share of category pages crawled weekly. After blocking the filter URLs, total crawl falls back, and new listings are crawled faster than before.

    Who should see which metrics

    Crawl metrics are team metrics. For leadership, a single line such as “new products are being discovered by Google within two days on average” is useful context; raw crawl charts are not. Keep the detailed metrics in the technical SEO dashboard, with alerts on sudden changes, and bring them to leadership only when they explain a business outcome or a risk.

    Watch for this:

    Unverified logs include spoofed Googlebot requests, which can make crawl volume look very different from reality. Verify against Google’s published IP ranges or by reverse DNS before calculating any crawl metric.

    Setting targets that make sense

    Targets for crawl metrics should come from your own baseline rather than external benchmarks, because crawl patterns vary widely by site size, update frequency and authority. Measure each metric for a couple of months, look at the normal range, and set targets that represent a meaningful improvement for the sections that matter. For example, a news site might aim to have new articles crawled within an hour, while a large catalogue might aim for new products within two days and every priority category within a week. For crawl waste, the target is usually a reduction after a specific fix, such as blocking a parameter pattern. Revisit targets after major site changes, because a migration or a new section resets what “normal” looks like.

    When crawl metrics do not matter much

    On small sites of a few hundred or a few thousand pages that change infrequently, Google usually crawls everything it needs. Crawl metrics are then mainly useful for spotting errors and incidents. Google’s own crawl budget guidance is aimed at very large sites and sites that change quickly, and smaller sites rarely need to manage crawl actively.

    Use crawl data as an indicator measured against goals, such as priority coverage, speed of discovery and crawl waste, on sites large or dynamic enough for crawling to be a constraint. Total crawl volume on its own belongs in diagnostics, not on a KPI list.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.