Discussion
  • Technical SEO

User-triggered Google fetchers such as NotebookLM generally ignore robots.txt. In log analysis, do you treat them as visits, as scraping, or ignore them?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • Niraj Raut
    Niraj Raut
    Original poster

    Treat them as their own category: neither crawling, nor human visits, nor scraping in the usual sense. Google’s documentation describes user-triggered fetchers, such as the Google-NotebookLM fetcher added in October 2025, as generally ignoring robots.txt because a user asked for the page to be fetched. They should not be counted in crawl metrics, since they do not reflect Google’s crawling decisions, and they are not sessions, since no person views the page in a browser. They are, however, a useful signal: each fetch shows that someone brought your page into an AI tool. Tracking them separately tells you which content people use with AI assistants, which is information you cannot get elsewhere.

    How to classify them in logs

    Category Examples How to use the data
    Search crawlers Googlebot, Bingbot Crawl analysis: frequency, coverage, errors
    AI training and search crawlers Declared AI crawlers Access decisions, load, AI visibility inputs
    User-triggered fetchers Google’s user-triggered fetchers such as NotebookLM; assistants’ user-requested fetches Signal of content use in AI tools
    Unverified or spoofed agents Requests claiming a known agent from unexpected IPs Security and load management

    Verify before you classify

    Google publishes IP ranges for its crawlers and fetchers, and those files moved to a new location in March 2026. Check requests claiming to be Google’s user-triggered fetchers against the published ranges or with reverse DNS, as you would for Googlebot. Spoofed user agents are common, and unverified requests should not be counted as genuine user-triggered activity.

    What the data can tell you

    • Which pages people bring into AI tools, which often differ from top search landing pages.
    • Which content types are used for research and summarisation, such as reports, guides and documentation.
    • Trends over time, as AI tool use grows.
    • Spikes after publications, campaigns or news coverage.

    Should you block them?

    Because they generally ignore robots.txt, blocking user-triggered fetchers means blocking at the server or CDN by user agent and verified IP. That denies users who deliberately asked an AI tool to read your page, which is usually not what a publisher wants, and it can seem hostile to people trying to engage with your content. Blocking can make sense for paywalled content that should not be extracted, or where fetch volume causes real load. For most sites, allowing them and tracking them is the better choice.

    Hypothetical example:

    A research consultancy notices requests from Google’s user-triggered fetchers in its logs and initially counts them as crawling, which inflates its crawl metrics. After separating them, it finds that most fetches are for its annual reports and methodology pages, which receive modest search traffic but appear to be widely used in AI research tools. The consultancy adds clear summaries and citation guidance to those pages, and starts reporting user-triggered fetches as an indicator of research use alongside search and referral data.

    Watch for this:

    Counting user-triggered fetches as human visits inflates audience figures, and counting them as crawling distorts crawl analysis. Keep them in a separate bucket with a clear definition.

    Other assistants’ user-triggered fetches

    Google is not the only company with user-triggered fetchers. Other AI assistants fetch pages when users ask them to read or summarise a URL, and their documentation describes separate user agents for this, distinct from their training and search crawlers. Apply the same approach: verify where possible, classify separately, and treat the data as a signal of how people use your content with AI tools. Comparing user-triggered fetches across assistants can also show which tools your audience uses most.

    Reporting

    A short monthly line in technical or content reporting is enough: verified user-triggered fetches by content type and top pages, with trends. Over time, it helps show how content is used beyond search and browsers, which is increasingly relevant for content strategy and for conversations about AI’s effect on traffic.

    Separate user-triggered fetchers from crawlers and from human visits, verify them against Google’s published ranges, use them as a signal of how people bring your content into AI tools, and block them only where there is a specific reason such as paywalled content or real load problems.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.