Discussion
  • Crawling
  • Indexing
  • Technical SEO

For parameter URLs, which has worked better at scale for you: robots.txt disallow that strands link signals, or crawlable noindex that keeps costing crawls?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    At scale, robots.txt usually wins for parameter patterns that have no search value, and Google’s own documentation leans the same way. The crawl budget guide says: “Don’t use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time”. Its faceted navigation guidance recommends disallowing faceted URLs you do not need indexed.

    The “stranded link signals” worry is real but usually smaller than it sounds. Parameter URLs rarely attract external links, apart from tracking and affiliate URLs, which need different handling anyway. Most of the stranded signal is internal, and that is a symptom of the site linking to worthless URLs in the first place. The better setups do not pick one directive for every parameter. They sort parameters by type, stop generating crawlable links to the worthless ones, block what remains with robots.txt, and keep noindex for transitions and edge cases.

    What each option actually costs

    Factor robots.txt disallow Crawlable noindex Crawlable with rel=canonical
    Crawl requests Stopped once Google has the updated file Continue; fewer over time per URL, but new combinations keep appearing Continue; may fall slowly as Google trusts the canonical
    Server load None from Googlebot Every crawl renders the page, often an uncached database query Same as noindex
    Index outcome URL can still be indexed without content if linked (“Indexed, though blocked by robots.txt”) Dropped once recrawled Usually folded into the canonical; it is a hint, not a rule
    Link signals Google cannot see the page’s links or canonical, so signals to these URLs do not consolidate anywhere Links followed at first; long-term noindex pages may lose that too Signals consolidate to the canonical when Google accepts it
    Speed of effect Fast for crawling; slow for URLs already indexed Slow: depends on recrawl of every URL Slowest to reduce crawling

    One detail changes how the noindex option looks. John Mueller said in December 2017 that a page left on noindex for a long time is eventually treated as noindex, nofollow, so its links stop being followed. The idea that noindex keeps link equity flowing indefinitely is weaker than it looks. You pay the crawl cost and may lose much of the link benefit anyway.

    How the pipeline treats each option

    Googlebot discovers a parameter URL through a link
    ↓
    Is the URL disallowed in robots.txt?
    ↓
    Yes: not fetched. Content, noindex and canonical are never seen. The URL may still be indexed from links alone.
    ↓
    No: fetched and processed. noindex drops it; canonical may cluster it with the clean URL; either way the crawl was spent.

    Two combinations cause most of the confusion. Disallow plus noindex does not work, because Google never fetches the page to see the noindex. Noindex plus a canonical pointing elsewhere sends conflicting signals (one says “drop this”, the other says “this duplicates that URL”), and Google representatives have advised against mixing them.

    Sort parameters by type before choosing a directive

    “Parameter URLs” covers several different problems. Google’s Gary Illyes and Martin Splitt said in early 2026 that faceted navigation (about 50%) and action parameters (about 25%) made up most of the crawling issues Google saw in 2025, well ahead of irrelevant parameters such as tracking codes. Those types need different treatment.

    Parameter type Search value External links What usually works
    Tracking and click IDs (utm_*, gclid, affiliate IDs) None Sometimes many Keep crawlable with a canonical to the clean URL. Do not disallow, because that strands those links. Avoid redirects that strip click IDs your ads or analytics depend on.
    Sort, view, items per page, display currency None Rare Remove crawlable links where possible; disallow the pattern
    Session IDs None Rare Fix at source (cookies); disallow in the meantime
    Action parameters (add to cart, wishlist, compare, print) None None Disallow; ideally use buttons or POST requests rather than links
    Filters with demand (brand plus category, colour plus type) Real Some Clean, indexable, self-canonical landing pages, linked with <a href> and listed in the sitemap
    Filters without demand, multi-select combinations None Rare No crawlable links; disallow the pattern
    Pagination (?page=2) Discovery of deeper items Rare Keep crawlable and self-canonical; do not block
    Internal search results (?q=) Usually none Occasional Disallow; turn proven queries into curated pages

    The root-cause fix sits upstream of both directives. If filters with no search value are rendered as buttons or form controls rather than crawlable links, Google never discovers most of those URLs, so there is little left to block and almost no internal signal to strand. Google’s faceted navigation guide also notes that rel="nofollow" only helps if every link to the URL carries it, and that a canonical may reduce crawling of non-canonical versions only over time. Neither is a substitute for not generating the links.

    Why the answer changes with scale

    Google’s crawl budget guide is aimed at very large sites (a million or more pages changing weekly) or medium sites (10,000 or more pages changing daily), and at sites with many URLs in “Discovered - currently not indexed”. Below that, a few thousand crawlable noindexed sort URLs rarely matter, and crawlable noindex or canonical is a reasonable, low-risk choice. Once the catalogue is large, facets multiply combinatorially. Three filters with ten values each can produce over a thousand combinations per category before sort and paging multiply them again, so noindex turns into a permanent crawl tax. That is where robots.txt earns its place.

    Hypothetical example:

    A retailer has 80,000 products, and logs show Googlebot has requested 2.5 million distinct parameter URLs in 90 days. Most are sort orders and multi-select filter combinations, all noindexed. Around a tenth of product pages have not been crawled in 30 days, and the Crawl Stats report shows average response time rising at peak. Here the noindex setup is costing product discovery and server capacity. The plan: build clean landing pages for the 300 filter combinations with measurable demand, render the rest of the filter controls without crawlable links, disallow sort and multi-select patterns, and keep canonicals on tracking parameters. The measure of success is product-page crawl frequency and time to index new products, not the number of excluded URLs.

    Switching from noindex to disallow without making a mess

    If the parameter URLs are already indexed, a straight switch to disallow freezes them in the index: Google can no longer see the noindex, so the URLs can linger as “Indexed, though blocked by robots.txt”. The textbook order is noindex first, wait until they drop out, then disallow. On a site with millions of URLs, that wait can take months because it depends on recrawling each one.

    A pragmatic alternative is to check whether the indexed parameter URLs actually appear for real queries. Filter the Search Console performance report to URLs containing the parameter. If they earn no meaningful impressions, disallowing straight away is usually acceptable: the stale entries mostly surface for site: searches, not for queries that matter. If some do earn impressions, give those patterns a clean indexable equivalent, or canonicalise them, before blocking.

    Watch for this:

    Broad patterns such as Disallow: /*? also block CSS and JavaScript files with version query strings, paginated listings, and API endpoints the page needs to render. Search Console cannot test a robots.txt file you have not published yet, so test patterns against a sample of real URLs with a robots.txt parser (such as Google’s open-source one) before release, then confirm with the robots.txt report and URL Inspection once it is live. Remember too that Google’s crawl budget guide warns against using robots.txt to temporarily reallocate crawl budget. Only block what you never want crawled.

    The evidence that should decide it for your site

    1. Server logs: the share of Googlebot requests going to each parameter pattern, the trend over time, and response times for those requests compared with product and category pages.
    2. Crawl freshness of important pages: how long new or updated products take to be crawled and indexed. This is the outcome parameter cleanup is meant to improve.
    3. Page indexing report: movement in “Indexed, though blocked by robots.txt”, “Blocked by robots.txt”, “Excluded by ‘noindex’ tag”, “Discovered - currently not indexed”, “Alternate page with proper canonical tag” and “Duplicate, Google chose different canonical than user”.
    4. Backlink data by parameter pattern: if a pattern has meaningful external links, canonicalise it rather than blocking it.
    5. Search Console performance for parameter URLs: any pattern earning clicks has search value and needs a clean landing page, not a block.
    6. Internal link counts from a crawl: how many links per template point at parameter URLs. A high number means the fix belongs in the template, whichever directive you choose.

    The result that would change the decision: if logs show parameter URLs take only a small share of Googlebot requests and important pages are crawled promptly, leave a working noindex or canonical setup alone. The engineering risk of a robots.txt change is not worth a problem you do not have.

    The decision rule I would use

    Keep crawlable, with a canonical, anything that has external links or search demand. Block with robots.txt anything that has neither and exists in volume, after removing as many crawlable links to it as the platform allows. Use noindex as a transition tool or for small, stable sets of URLs, not as the long-term control for a facet system that keeps generating new combinations. Then judge the change by product crawl frequency and time to index, because a shrinking count of excluded URLs in Search Console says little about whether Google is now spending its time on the pages you sell from.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.