Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- Indexing
- Technical SEO
When the Googlebot smartphone to desktop ratio shifts suddenly in logs, have you ever tied it to an indexing or ranking change, or is it usually noise?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Most sudden shifts in the ratio of Googlebot smartphone to desktop requests turn out to be noise or changes on Google’s side, and I would not act on one unless it comes with something else: errors or redirects served to one crawler type, a difference between what the mobile and desktop versions serve, or a drop in crawling of important pages. Google indexes sites with its smartphone crawler, and desktop crawling continues at a lower level for other purposes, so the balance between the two can move without any consequence for rankings. A ratio shift is a reason to check parity and status codes, not a finding in itself.
Background: what Google crawls with and why
Google completed its move to mobile-first indexing some time ago, and since 2024 it has said it crawls and indexes sites with its smartphone crawler, with sites that do not work on mobile at risk of not being indexed. That means the smartphone crawler is the one whose view of your site determines what gets indexed. The desktop crawler has not disappeared; Google still uses it for some fetches. So a site will normally see a large majority of verified Googlebot requests from the smartphone crawler, with a smaller and variable desktop share. The precise split varies by site and over time, and Google does not document what the desktop fetches are used for in detail. That uncertainty is why a change in the desktop share, on its own, tells you very little.
Why the ratio moves
- Google’s own processes. Google uses its desktop crawler for various purposes alongside the smartphone crawler used for indexing, and the mix changes as those processes change.
- Your site’s behaviour by user agent. If you serve different HTML, redirects or status codes to mobile and desktop user agents, a change on your side can shift how much each crawler fetches.
- Unverified bots. Spoofed Googlebot user agents are common. If your analysis does not verify requests, a scraper using a desktop Googlebot string can distort the ratio.
- Logging and caching changes. A CDN that caches differently by user agent, or a change in what gets logged, alters the counts without Google changing anything.
- Crawl changes for other reasons. New sections, sitemaps or redirects can increase crawling by one type for a period.
When it might matter
Pattern Possible meaning Check Smartphone requests fall and errors to smartphone rise Mobile version failing or blocked for the indexing crawler Status codes and response sizes by crawler type; URL Inspection Desktop share rises after a deployment that changed user-agent handling Different content or redirects by user agent Fetch key URLs as each crawler and compare Ratio shifts only on one section Section-specific behaviour, such as a separate mobile template Compare that section’s mobile and desktop responses Ratio shifts across the whole site with no errors and no deploy Likely Google-side or measurement noise Verify bots; confirm with Crawl stats “By Googlebot type” A diagnostic sequence
- Verify the requests. Filter to requests from Google’s published IP ranges or confirmed by reverse DNS. Recalculate the ratio. Many “shifts” disappear at this step.
- Check Search Console’s Crawl stats report by Googlebot type for the same period. If it shows no shift, your logs are probably capturing something else.
- Segment by status code and response size for each crawler type. A rise in errors or a drop in average response size for smartphone requests is the pattern that deserves attention.
- Check parity. For a sample of important URLs, fetch as smartphone and desktop Googlebot and compare the HTML: content, links, structured data, canonicals and robots directives.
- Line it up with deployments and CDN or configuration changes.
- Check search data. If clicks and impressions for the affected pages are stable, the shift is almost certainly not affecting ranking.
Watch for this:
Sites using dynamic serving, where the same URL returns different HTML to mobile and desktop user agents, are the ones where a ratio shift is most likely to matter. If the mobile HTML is missing content, links or structured data that the desktop HTML has, Google indexes the thinner version. Parity problems are worth fixing whether or not the ratio moved.
Hypothetical example:
A retailer’s logs show desktop Googlebot rising from a small share of requests to nearly half over a week. The team worries about a change in indexing. After verification against Google’s IP ranges, most of the extra “desktop Googlebot” requests turn out to come from a scraper using a Googlebot user agent. The verified ratio has barely moved, Crawl stats confirms it, and search performance is stable. The team blocks the scraper at the CDN and adds bot verification to its standard log reports.
What to monitor routinely
- Verified Googlebot requests by type, status code and average response size, weekly.
- An alert if error rates for the smartphone crawler rise above normal.
- A monthly parity check on a sample of key templates, comparing mobile and desktop responses.
- Deployment annotations on crawl charts, so shifts can be matched with changes.
Verify first, then look for errors and parity differences. If both are clean and search data is stable, a change in the smartphone to desktop ratio is noise worth noting rather than a problem worth solving.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.