Discussion
  • AI Search
  • Search Console

Search Console now has an AI features opt-out while Bing relies on meta tags like noarchive. How are you keeping AI controls consistent across engines?

Published 1 reply

Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • Niraj Raut
    Niraj Raut
    Original poster

    Start from a single policy, written by purpose, and then implement it through each platform’s own mechanism, accepting that the controls do not line up exactly. Google now offers a Search generative AI control in Search Console, rolled out to all sites worldwide on 31 August 2026, which lets a site opt out of AI Overviews, AI Mode and AI Overviews in Discover at site level; Google says the setting is not used as a ranking signal outside those features, but opted-out sites receive no traffic or impressions from them. Google-Extended separately controls whether content is used to train Gemini models. Bing has documented that robots meta directives such as noarchive and nocache limit how content is used in its AI answers, which works page by page. Other AI companies control access through their crawlers in robots.txt. Consistency comes from the policy and its documentation, not from one switch.

    Write the policy by purpose first

    Purpose Question to decide
    Traditional search results Almost always allowed
    AI answers in search engines Do we want our content used and cited in AI summaries and AI modes?
    AI assistants’ search features Do we want to be cited in assistants that search the web?
    Model training Do we allow our content to be used to train models?
    User-triggered fetches Do we allow assistants to fetch pages when users ask?

    Map the policy to each platform’s controls

    • Google search AI features: the Search generative AI control in Search Console, site level.
    • Google model training: the Google-Extended token in robots.txt.
    • Bing and Microsoft’s AI answers: robots meta directives as documented by Bing, applied per page or template.
    • OpenAI and other AI companies: their documented crawler user agents for training, search and user-triggered fetching, controlled in robots.txt and, where needed, at the CDN.
    • CDN-level controls: provider options for AI crawlers, such as Cloudflare’s September 2026 option aimed at disallowing training for mixed-use crawlers, checked carefully for side effects on search.

    Where the controls do not match

    Google’s search AI control is site-wide, while Bing’s directives work page by page, so a policy that treats sections differently, for example allowing AI answers for help content but not for premium articles, can be implemented precisely on Bing and only as all-or-nothing in Google’s search features. Some controls affect both AI answers and other uses; some platforms document their behaviour in detail and others less so. Record these mismatches in the policy document, with the practical outcome for each platform, so stakeholders understand why behaviour differs between engines.

    Hypothetical example:

    A publisher decides to allow AI answers that cite and link to its free articles, but not to allow training use, and to keep premium articles out of AI answers where possible. It leaves Google’s search AI features enabled, since the setting is site-wide and free articles make up most of the site, and disallows Google-Extended. It applies Bing’s restrictive directives to the premium section only, disallows known training crawlers from other AI companies while allowing their search crawlers, and documents each choice in a one-page policy with a table of platforms, controls and effects. When a new AI crawler appears, the team adds it to the table using the same purpose-based rules.

    Watch for this:

    Blanket blocking at the CDN, or directives applied sitewide without checking what they affect, can remove content from traditional search or from AI search products you wanted to be in. Test changes on a section first, and monitor search traffic and AI referrals after every change.

    Communicate the policy internally

    Different teams make decisions that affect AI access without realising it: security teams change bot rules, developers edit robots.txt, content teams add directives to templates, and legal teams negotiate licensing. A short, shared policy with named owners prevents conflicting changes. Include it in change checklists for CDN rules, robots.txt and templates, and make sure legal and commercial teams know which controls exist before discussing licensing with AI companies, since the technical position often shapes what can be offered.

    Keep it maintained

    AI controls are changing quickly: new crawlers, new platform settings, changes to what existing controls cover. Assign an owner for the AI access policy, review it quarterly against platform documentation, and log every change with its date. Use Search Console’s generative AI reports, Bing’s AI Performance report, analytics referrals and server logs to see the effect of each decision.

    Decide the policy by purpose, implement it through each platform’s own controls, document where controls differ in scope, test changes before applying them sitewide, monitor the effects, and review the policy regularly as platforms change.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.