Discussion starter: an ecommerce SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- eCommerce SEO
- Indexing
Internal site search pages that rank and convert: whitelist a few for indexing, or replace them with curated landing pages?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Treat the search pages that rank as research, and give the demand they have found a proper landing page. For the proven queries, I would build curated pages (an existing category, an indexable filter page or a new collection), 301 the search URLs to them, and keep every other search URL out of the index with
noindex. A tightly controlled allowlist (the whitelist in your question) is a defensible alternative when the valuable queries are too many and too long-tail to curate, or when the platform makes new landing pages slow to ship.The deciding factor is where each page’s value comes from. If a search page converts because a clear query returns a specific, relevant set of products, you can reproduce that set on a page you control, with stable content and merchandising. If the value is spread across hundreds of small queries that only a search engine can match, curation will not keep up, and an allowlist with quality gates is the more realistic option. The two can coexist: curated pages for the head,
noindexfor the rest of search, and an allowlist only where the tail is large and demonstrably profitable.What Google has said about indexing site search pages
Google has not banned indexed search results. In April 2023, John Mueller said that if you cannot select which search result pages should be indexable, you should block all of them from being indexable, using a robots.txt disallow or a
noindexrobots meta tag. He added that a search page returning no results had no reason to be indexable, and that open search result pages were still regularly used to spam Google’s results, something that took little work to prevent and was a hassle to clean up afterwards.Read that the right way round. Choosing a few search pages for indexing fits the advice; leaving the whole search space open does not. An allowlist is only safe if code enforces the selection, so that every URL outside it carries
noindexby default.Why a ranking search page is a fragile asset
- The content changes without anyone deciding. Stock changes, relevance tuning, synonym rules and new products can all swap the products on a page that ranks, including the ones that converted.
- Merchandising is awkward. Pinning hero products, excluding low-margin lines or adding buying guidance on a search template can need workarounds that nobody maintains.
- It can duplicate a page you already have. “Linen sheets” as a search URL and a linen filter on the sheets category serve one intent from two URLs.
- The title comes from user input. The template that writes a good title for “linen sheets” writes a spam title for whatever someone puts in the query string. That is why non-allowlisted and empty search pages need
noindex. - Thin results read as thin pages. A search page with no results, or next to none, can be reported as “Soft 404” in the Page indexing report, and it gives searchers little reason to stay.
Allowlist or curated page: how the options compare
Condition Lean towards an allowlist Lean towards curated landing pages Proven queries Hundreds of long-tail queries, each small A few to a few dozen queries carrying most of the revenue Overlap with existing pages Queries match nothing in the category or filter structure Queries map to a category, a filter combination or an obvious new collection Result stability Search relevance is good and results change slowly Results shift with stock or relevance tuning Merchandising needs Low: any relevant product set converts High: margin, brand or stock control matters Platform Landing pages are slow or costly to create Collections or categories are cheap to create and redirect to Resourcing Engineering time for normalisation and allowlist logic Merchandiser time to build and maintain pages Current exposure Search is already noindexby default, with allowlist logic in placeSearch pages are open, and spam or empty results are already indexed Evidence to gather before choosing: the search URLs with clicks in Search Console over the last 12 months and their queries, revenue by landing page in analytics, how many search URLs sit in the Page indexing report and whether any carry spam text, and a weekly snapshot of the products each proven search URL returns. If the product set churns heavily over a month, the page that earned the ranking is not the page searchers are getting now, and that favours curation.
If you allowlist, the controls that make it safe
- The allowlist lives in code or configuration. A search URL is indexable only if its normalised query is on the list; everything else gets
noindex. - One URL per query. Normalise case, spacing, encoding and parameter order, strip sort and view parameters, and point the canonical at the normalised URL.
- A results floor. Below a minimum product count the page falls back to
noindex, and zero results meansnoindex. - A written title and H1 for each allowlisted query instead of the raw query string.
- Internal links from relevant categories or a “popular searches” module, and inclusion in the XML sitemap. Keep non-allowlisted search URLs out of both.
- The query is escaped in page output, because it is user-controlled input.
- A monthly review: drop queries whose revenue has faded, and promote the strongest to curated pages.
Noindex versus robots.txt for the rest of search
For the non-allowlisted search URLs,
noindexis the reliable way to get them out of the index, and Google has to crawl a page to see it. A robots.txt disallow stops crawling but does not remove URLs Google already knows about. Blocked URLs can stay indexed without their content, which shows up in the Page indexing report as “Indexed, though blocked by robots.txt”. Once thenoindexis processed, the same report lists them under “Excluded by ‘noindex’ tag”.Search Engine Journal’s July 2026 write-up of a case where Shopify search pages were indexed despite a robots.txt block recommends
noindexfor the same reason: robots.txt controls crawling, not indexing. The case also carries a gotcha worth checking in your own file. Mueller’s explanation, quoted in the same article, was that “the more specific rules win, so if you have a user-agent: Googlebot section, it will only use that section”. If someone adds a Googlebot group for an unrelated reason, the search disallow in the general group no longer applies to Googlebot.On a very large store where search URLs take a noticeable share of Googlebot requests in the Crawl stats report or server logs, some teams add a disallow after the
noindexhas been processed and the URLs have dropped out. That saves crawling, but it reopens the risk of blocked URLs being indexed without content if they are linked from elsewhere, so recheck the Page indexing report afterwards. On a store with a few thousand products, crawl capacity is unlikely to be the reason to act: Google says its crawl budget guide is aimed at large sites (over a million unique pages changing about weekly), medium sites (over 10,000 pages changing daily) and sites with many URLs in “Discovered - currently not indexed”. For smaller stores, index quality and spam are the reasons.Replacing search pages without losing what made them work
- Inventory: export every search URL with organic clicks in the last 12 months, with its queries, clicks and landing-page revenue.
- Map each query: to an existing category, an existing indexable filter URL, a new curated page, or retirement.
- Record what converted: which products sold from each search URL, the price band and brands. That becomes the brief for the curated page’s product set.
- Build the page: a title and H1 that keep the query wording that earned the clicks, a product set you control, and copy only where it helps people choose (sizing, material differences), not filler.
- Redirect: 301 each search URL to its new page. Redirect rules that match query strings need testing; once the old URL is recrawled, URL Inspection should report it as “Page with redirect”.
- Link it: from the parent category and any relevant navigation or “popular searches” module.
- Close the rest:
noindexon every remaining search URL, including a rule for zero results. - Measure by query group: clicks and revenue for the migrated queries before and after. Recrawl timing varies by URL, so judge each page from the date URL Inspection shows the new page indexed, not from the release date.
Hypothetical example:
A homewares store finds 60 search URLs with organic clicks over the past year. Eight of them bring most of the search-page revenue. Three map to existing categories, so those search URLs redirect there and the categories surface the filters people were searching for. Two map to filter combinations the store already lets Google index. Three need new curated collections, such as “housewarming gifts under $100”. The other 52 get
noindexwith the rest of search. The store skips the allowlist entirely, because its tail is not worth the engineering.Watch for this:
Do not redirect a search URL to a page that does not satisfy its query. If “waterproof hiking boots wide fit” redirects to a general hiking boots category, the ranking may not transfer, and the conversion rate that justified keeping the page will not transfer either. Build the narrower page or leave that URL alone until you can.
If this were my store, I would move the proven head queries to curated pages,
noindexthe rest of search, and build an allowlist only if the long tail is too large and too profitable for curation to keep up. After that, the queries people type into your site search and the search URLs Google tries to rank stop being an index-management problem and become a steady source of landing-page ideas.Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.