Discussion
  • Crawling
  • Search Console
  • Technical SEO

Search Console Crawl Stats and raw server logs often disagree on volumes and response times. Which do you trust when they conflict, and why?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    They disagree because they measure different things, so the right question is which one answers the question you are asking. Search Console’s Crawl stats report shows Google’s view: every request Google’s crawlers made to the host, including resources such as CSS, JavaScript and images, and response times as Googlebot experienced them, network included. Origin server logs show your view: requests that reached your server, with server-side processing time. For crawl volume, trust verified logs taken at the edge, or Crawl stats if you do not have them. For response time, trust Crawl stats to tell you what Google experienced, which is what drives its crawl rate decisions, and trust your logs to tell you where the time was spent.

    Why the numbers differ

    Cause Effect
    CDN caching Requests served from the CDN never reach the origin, so origin logs undercount crawl volume
    Resources included Crawl stats counts CSS, JavaScript, images and other resources; log analyses often filter to pages only
    Different Google crawlers Crawl stats covers Google’s crawler types; log filters may include only Googlebot smartphone and desktop
    Spoofed user agents Unverified logs include fake “Googlebot” requests; Crawl stats does not
    Response time definition Crawl stats measures time to fetch from Google’s side, including network; server logs often record processing time only
    Time zones and day boundaries Search Console reports in Pacific Time; your logs may use UTC or local time
    Host scope Crawl stats is per host within the property; logs may combine hosts or cover only one

    Reconciling them

    1. Use edge or CDN logs where possible, so cached responses are included.
    2. Verify Googlebot by Google’s published IP ranges or reverse DNS, and exclude everything else.
    3. Include all Google crawler user agents and all resource types when comparing volumes.
    4. Align time zones and compare over whole days or weeks.
    5. Compare host by host. A domain property’s Crawl stats shows hosts separately; make sure your logs cover the same hostname.

    After these steps the volumes usually come much closer. Remaining gaps tend to come from edge cases such as requests blocked by a firewall before logging, or logging gaps during incidents.

    Response times: two useful numbers

    A slow average response time in Crawl stats with fast server processing times in your logs usually means the time is being lost outside your application: at the CDN, in TLS negotiation, in network routes, or in large response sizes. A slow time in both points to the application or the database. The Crawl stats figure matters for crawling because Google adjusts how much it crawls based on how the host responds; if responses slow down, Google may crawl less. Your logs tell you what to fix.

    What Crawl stats is good for, and where it is limited

    • Good for: totals by response code, file type, purpose (discovery or refresh) and crawler type; host status, including robots.txt fetch problems and DNS or connectivity issues; trends over the last 90 days.
    • Limited for: URL-level analysis, since it shows only example URLs; longer history; joining with your own data.

    Logs are the only source for URL-level crawl frequency, for analysis beyond 90 days if you retain them, and for joining crawl data with page type, revenue or content attributes.

    Hypothetical example:

    A retailer’s origin logs show a fraction of the crawl requests that Crawl stats reports, and a much faster average response time. The team switches to CDN logs, verifies Googlebot and includes image and script requests; the volumes now match closely. The response time gap remains: Crawl stats averages several hundred milliseconds more than server processing time. Investigation shows that product pages are not cached at the edge for Googlebot because of a cookie rule, so every request travels to the origin across regions. Fixing the cache rule reduces Crawl stats response times, and Googlebot’s daily requests to product pages rise over the following weeks.

    Host status deserves its own attention

    One part of Crawl stats has no equivalent in most log setups: host status. It shows whether Google had problems fetching robots.txt, resolving DNS or connecting to the server over the last 90 days. Google has said it may pause or reduce crawling when robots.txt returns server errors, so a robots.txt problem can quietly limit crawling of the whole host. Your logs may not show it at all if the requests failed before reaching the server, for example because a firewall or CDN rule blocked them. Check host status whenever crawl volume falls unexpectedly, and set up an external monitor that fetches robots.txt regularly and alerts on anything other than a successful response.

    Keep both, for different jobs

    • Check Crawl stats weekly for host status, error spikes and response time trends.
    • Keep verified edge logs, or at least a daily summary per URL, for crawl frequency and page-level analysis.
    • When the two disagree sharply without an explanation, investigate: it often reveals a caching, routing or bot-verification problem worth fixing.

    Use Crawl stats for what Google experienced and for host health, use verified edge logs for volumes and URL-level detail, and treat a large unexplained gap between them as a finding, not a nuisance.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.