Discussion
  • Enterprise SEO
  • Indexing

Personalisation and A/B tools change content by segment. Have you had cloaking worries or indexing issues, and what rules did you set for bots?

Published 1 reply

Discussion starter: an enterprise SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Personalisation and A/B testing are compatible with search as long as Googlebot is treated like an ordinary anonymous visitor and not given special content. Google’s guidance on website testing says testing is fine, but you should not cloak by deliberately showing Googlebot different content from users, you should use rel canonical on variant URLs pointing to the original, use 302 rather than 301 redirects for redirect tests, and run tests only as long as necessary. For personalisation, the default experience that an anonymous visitor sees should be complete and representative, because that is what Googlebot will index. Problems usually come from tests and segments that hide important content from the default, or from rules that exclude bots in ways that amount to cloaking.

    Rules that keep testing safe

    • No user-agent targeting. Do not show Googlebot the control while users see variants, or the reverse. Let Googlebot fall into tests like any visitor, or be served the default.
    • Canonical on variant URLs. If a variant lives on a different URL, it should point its canonical to the original.
    • Temporary redirects for redirect tests. Use 302s, not 301s, so the original stays canonical.
    • Limited duration. End tests when they reach a decision; do not leave variants running for months.
    • A complete default. The non-personalised experience should include all the content and links you want indexed.

    Where personalisation creates search problems

    Pattern Problem
    Content shown only to logged-in or returning users Googlebot never sees it
    Geo-personalisation that replaces content by country Googlebot mostly crawls from the US and indexes the US variant; other markets’ content is missed unless it has its own URLs
    Segment-specific navigation Internal links Googlebot does not see, weakening discovery
    Personalised titles or headings Inconsistent signals between crawls
    Heavy client-side testing scripts Flicker, layout shift and slower interaction, affecting users and page experience metrics

    A governance model that works

    1. SEO review for tests that touch search-critical elements: titles, headings, main content, internal links, canonicals, structured data and indexability.
    2. A register of running tests and personalisation rules, so anyone investigating a ranking change can check what was live.
    3. Default-first design: personalisation adds to or reorders content for segments; it does not remove important content from the default.
    4. Separate URLs for genuinely different markets, with hreflang, rather than swapping content on one URL by location.
    5. Performance budgets for testing tools, so scripts do not degrade the experience for everyone.

    Hypothetical example:

    A software company personalises its homepage by industry, detected from the visitor’s company IP. Anonymous visitors, including Googlebot, see a generic version with no industry sections and no links to industry pages. Those industry pages are poorly linked and rank weakly. The company changes the design so the default homepage includes a short section linking to every industry page, and personalisation reorders sections for recognised visitors rather than hiding them. Industry pages gain internal links from the homepage, their crawl frequency rises, and their rankings improve over the following months.

    Watch for this:

    Some testing tools let you exclude bots from experiments, which then serves bots the control indefinitely. That is not deliberate cloaking, since users also see the control, but if the winning variant launches and the tool keeps bots on an old control, the gap between what users and Googlebot see can grow. Check what bots receive after every launch.

    Testing SEO-critical elements themselves

    Sometimes the test is about search: new title formats, different content structures, changes to internal linking. Those tests are better run as SEO split tests on groups of pages than as user-level A/B tests on one page. In an SEO split test, each page is either in the control or the variant, and every visitor, including Googlebot, sees the same version of that page. That avoids any question of cloaking and lets you measure the effect on search traffic, which user-level tests cannot do because search engines see only one version of each URL at a time. Use user-level A/B tests for conversion questions and page-level split tests for search questions, and keep the two separate so their results do not contaminate each other.

    How to check what Googlebot sees

    Use URL Inspection’s live test to see the rendered page Google receives, and compare with what an anonymous visitor in a clean browser sees. Crawl key templates with a tool set to Googlebot’s user agent and with a normal browser user agent, and compare content, links and metadata. Differences that are not explained by random test assignment deserve investigation.

    Treat Googlebot as an anonymous visitor, make the default experience complete, follow Google’s testing guidance on canonicals, redirects and duration, put search-critical tests through SEO review, and give genuinely different markets their own URLs rather than personalising one.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.