Discussion
  • AI Search

Has blocking GPTBot for training while allowing OAI-SearchBot kept you in ChatGPT answers? Any citation changes after splitting them?

Published 1 reply

Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Splitting the two is the right approach if you want to stay visible in ChatGPT search while opting out of training, and it matches how OpenAI documents its crawlers: GPTBot is used for training, OAI-SearchBot for ChatGPT’s search features, and a separate user agent for fetches a user requests during a conversation. Allowing OAI-SearchBot keeps your pages eligible to be surfaced in ChatGPT search. What you give up by blocking GPTBot is future inclusion in training data, which affects what the model “knows” about you when it answers without searching. Whether citations change after splitting is hard to attribute, because ChatGPT search also depends on third-party search providers and on how prompts are rewritten into searches.

    What each OpenAI crawler does

    User agent Purpose Effect of blocking
    GPTBot Collecting content that may be used to train models Your content is excluded from future training
    OAI-SearchBot Crawling for ChatGPT’s search features Your pages are less likely to appear as sources in ChatGPT search
    ChatGPT-User Fetching a page because a user asked ChatGPT to look at it User-triggered; such fetchers generally do not follow robots.txt in the same way as crawlers

    Two kinds of ChatGPT answer

    It helps to separate answers that search the web from answers that come from the model’s own knowledge. For searched answers, OpenAI’s help documentation says ChatGPT search uses third-party search providers and rewrites prompts into targeted queries. Your visibility there depends on being found by those searches and on OAI-SearchBot being able to access your pages. For answers from the model’s knowledge, what the model learned during training matters. Blocking GPTBot has no effect on content already in past training data, but over time newer models will know less about you from your own site and more from what others say about you.

    What to check if you split them

    • Your robots.txt has separate groups for GPTBot and OAI-SearchBot, with the intended rules in each.
    • Your CDN or firewall is not blocking OAI-SearchBot anyway. Several CDN providers now offer AI-crawler controls, and some block AI crawlers by default for new sites; Cloudflare, for example, added an option in September 2026 aimed at disallowing training for mixed-use crawlers.
    • Bingbot and other search engine crawlers are allowed, because ChatGPT search relies partly on third-party search providers.
    • Your logs show verified OAI-SearchBot requests after the change.

    How to monitor citations

    1. Referrals: ChatGPT adds a utm_source=chatgpt.com parameter to many links, and GA4’s AI Assistant channel captures ChatGPT referrals. Track sessions and landing pages before and after the change.
    2. Logs: OAI-SearchBot and ChatGPT-User activity shows which pages ChatGPT is fetching.
    3. Prompt tracking: run a fixed set of prompts in ChatGPT with search enabled and record whether you are cited, before and after.

    Expect noise. ChatGPT answers vary run to run, and model updates change behaviour independently of your robots.txt. A drop in citations in the week after you block GPTBot is more likely to be coincidence than cause, unless OAI-SearchBot was also blocked by mistake.

    Watch for this:

    The most common problem with split rules is an accidental block. A broad firewall rule for “AI bots”, a CDN default or a robots.txt wildcard can catch OAI-SearchBot along with GPTBot. If ChatGPT referrals fall sharply after a change, check access for OAI-SearchBot before looking for other explanations.

    Hypothetical example:

    A publisher blocks GPTBot and allows OAI-SearchBot in robots.txt. ChatGPT referrals continue at their usual level for a month, then fall by half. Logs show OAI-SearchBot requests stopped on the day the security team enabled a CDN rule blocking “AI crawlers”. The publisher adds an exception for OAI-SearchBot, verified by IP, and referrals return to their previous level within a few weeks. The robots.txt split was never the problem; an overlapping rule was.

    Other AI providers use similar splits

    OpenAI is not the only company separating training from search and user-triggered fetching. Google uses the Google-Extended token to let sites control whether content is used for Gemini training, separately from Googlebot, and blocking it does not affect ranking in Google Search. Other AI companies publish their own crawler names for training, search and user-requested fetches. The pattern is the same: decide separately whether you want to allow training, search indexing and user fetches for each provider, check each provider’s current documentation for the exact user agents, and review the rules periodically, because the list of crawlers and their purposes keeps changing. A single rule that blocks “all AI” is simple, but it usually removes you from AI search features you may want to be in.

    The trade-off to decide consciously

    Blocking training is a legitimate choice, often driven by licensing, copyright or commercial concerns. The cost is that future models will learn about your brand and subject matter from other sources rather than from you. For publishers whose business depends on content licensing, that cost may be worth paying. For brands whose goal is to be described accurately, allowing training while protecting premium content by other means may be the better balance. Either way, keep the search crawler allowed if you want to appear in ChatGPT’s searched answers.

    Split them if you want to opt out of training, make sure OAI-SearchBot and search engine crawlers are genuinely allowed at every layer, measure citations through referrals, logs and prompt tracking, and do not over-read short-term changes.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.