Discussion starter: a programmatic SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Indexing
- Technical SEO
Before a programmatic page becomes indexable, do you require minimum inventory or data? What threshold worked for you?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Yes, gate indexability, but do not borrow a fixed number from anyone. A threshold like “at least five listings” is a reasonable opening guess for many templates and a poor rule if it is the only rule. What makes a programmatic page worth indexing is whether it gives a searcher something the parent page, the sibling pages and the rest of the web do not. Inventory count is one proxy for that. Overlap with neighbouring pages, evidence of demand, supporting data and stability over time are the others.
The threshold that holds up is the one you derive from your own template: bucket existing pages by inventory, see where indexing rates, impressions and conversions per page change, set the gate there, and add hysteresis so pages do not flap in and out of the index as stock moves. The right number for a job board, a property portal, a local services directory and a product-attribute page on a store can differ widely.
What Google documents, and what it leaves to you
Google does not publish a minimum inventory for indexing. What it documents is the failure cases, which tell you what the gate is protecting against:
- Scaled content abuse is defined in the spam policies as “when many pages are generated for the primary purpose of manipulating search rankings and not helping users”. One listed example is “Scraping feeds, search results, or other content to generate many pages… where little value is provided to users”.
- Doorway abuse includes “Having multiple domain names or pages targeted at specific regions or cities that funnel users to one page”, which is exactly what location templates drift into when every suburb page shows the same inventory.
- Soft 404s: Google’s HTTP status documentation says a page returning a 2xx code with an empty page or an error message gets reported as a soft 404. A zero-result programmatic page served as a normal 200 fits that description.
- Rendering: Google’s JavaScript SEO documentation notes that a noindex in the original HTML can stop rendering, so removing it with JavaScript may not work. That matters if your pages load inventory client-side and flip noindex off once data arrives.
- Crawl budget: Google’s crawl budget guide is aimed at sites with 1M+ pages changing about weekly, 10k+ pages changing daily, or many URLs in “Discovered - currently not indexed”. Programmatic builds reach those conditions quickly.
Everything beyond that, including any specific number, is practitioner judgement. Treat anyone’s “five listings worked for us” as a data point about their template, not yours.
Why a count on its own is a weak gate
Two hypothetical pages show the problem. “Plumbers in Suburb A” has 12 listings, but they are the same 12 that appear on “Plumbers in Suburb B” and on the regional parent, because the site matches on a 20 km radius. “Plumbers in Remote Town” has three listings, and they are the only three plumbers who serve that town anywhere online. The count says index the first and hold back the second. Usefulness says the opposite.
Radius matching is a common way for count thresholds to get fooled. It inflates inventory on every nearby location page and produces the city-funnel pattern the doorway policy describes. Attribute pages on stores do the same thing: “red running shoes size 9” may show 15 products, 14 of which also appear on “red running shoes”.
The gates I would combine
Unique inventory
Count only live, in-scope items whose own location or attributes match the page. Exclude expired, duplicated and radius-only matches.
Overlap with parent and siblings
Measure the share of items that also appear on the parent or nearest sibling page. High overlap means the page adds a filter, not an answer.
Demand
Evidence that people search for this combination: impressions on similar existing pages, internal site search logs, keyword data or paid search query reports.
Supporting data
Content that makes the page useful even with few items: local price ranges, typical lead times, regulations, reviews, availability patterns. Three listings plus real local data can beat twenty listings and nothing else.
Stability
How long the page has stayed above the gate. Inventory that appears for two days and disappears creates index churn and wasted crawling.
How to find your own threshold
- Take the template’s existing pages, or a pilot batch released as indexable, and bucket them by unique inventory (for example 1, 2 to 4, 5 to 9, 10 to 24, and 25+).
- After at least 8 to 12 weeks, measure each bucket: share indexed (the Page indexing report or a URL Inspection API sample, watching “Crawled - currently not indexed” and “Discovered - currently not indexed”), impressions and clicks per page, conversions per page, and the share of pages with zero impressions.
- Look for the point where indexing rate and clicks per page step up. That is your first candidate threshold. If there is no clear step, inventory count is probably not the variable that matters for this template, and overlap or demand deserves more weight.
- Apply the overlap and demand gates to pages above the threshold. Pages that pass the count but fail both are still candidates to hold back.
- Validate with a holdout. Release a random sample of just-below-threshold pages as indexable, keep a comparable sample noindexed, and compare after a quarter. If the released pages earn meaningful clicks without pulling down the template’s overall indexing rate, the threshold is too strict. If they mostly sit in “Crawled - currently not indexed”, the threshold is not too strict; check the bucket just above it before deciding whether it is too loose.
- Revisit quarterly. Inventory, demand and Google’s treatment of the template all move.
Add hysteresis so pages do not flap
A single cut-off creates churn when inventory hovers around it: indexable on Monday, noindex on Thursday, indexable again next week. Each flip needs a recrawl to take effect, so Google’s view of the page lags reality and may never settle. A thermostat solves the same problem with a dead band: heating on at 19 °C, off at 21 °C, so it is not switching every few minutes. Apply the same idea with separate entry and exit thresholds, plus a time condition.
Hypothetical rule set:
A location and category page becomes indexable when it has at least 8 unique in-area listings for 14 consecutive days and less than 70% overlap with its parent. It returns to noindex only when unique listings stay below 4 for 30 consecutive days. Pages with zero listings for 90 days return a 404, unless they have earned links or steady impressions, in which case they stay live with supporting data and links to nearby alternatives. These numbers only illustrate the shape of the rule; the real values should come from the bucket analysis above.
Implementation details that decide whether the gate works
- Put noindex in the server-rendered HTML. Do not rely on JavaScript to add or remove it.
- List only indexable pages in XML sitemaps, and add them when they cross the gate.
- Do not link to millions of sub-threshold pages from crawlable navigation. Google still has to crawl a noindexed URL to see the noindex, so you spend crawl on pages you have already decided against. If a sub-threshold page does not need to exist for users, do not generate it.
- Do not use a canonical to the parent as the gate. A canonical is a hint for duplicate or near-duplicate pages; a filtered page with different items is not a duplicate, and Google may ignore the hint, leaving you without the control you thought you had.
- Avoid empty 200 pages. A zero-result page should return a 404, or be a noindexed page with useful alternatives if users reach it through internal navigation.
- Keep robots.txt for crawl control of infinite parameter spaces, not as the indexing gate. Blocked URLs cannot show Google a noindex, and they can still be indexed without their content if linked.
How strict to be depends on scale and site strength
Condition Lean towards a stricter gate Lean towards a looser gate Possible URL combinations Millions (location × category × attribute) Hundreds to low thousands Inventory volatility Items expire in days (jobs, rentals, events) Stable (businesses, long-life products) Supporting data per page Little beyond the listings Rich, page-specific data Current index health Many template pages in “Crawled - currently not indexed” Most template pages indexed and earning impressions Site strength in the vertical New or weak domain Established and trusted in the niche Commercial value per page Low High: each indexed page supports leads or sales Current index health is the most useful live signal. The documented part is simple: Google crawls and then chooses not to index some pages. The reasoned part is that if Google is already declining a large share of a template, releasing more pages below the same quality line is unlikely to change that decision and more likely to spread crawling thinner. That is an inference from how the reports behave, not a rule Google has published, so the holdout test above is how you confirm it on your own site.
The distinction that matters:
An inventory threshold answers “does this page have enough stuff?” The question Google’s systems are effectively answering is “is this page a better result than what already exists, including your own parent page?” Count is the cheapest approximation of the second question, which is why it needs overlap and demand checks beside it.
Where I would start
If I were launching a template today, I would begin with a conservative gate (unique in-area inventory, low overlap with the parent and some evidence of demand), release a pilot batch, and let the bucket analysis show where to move the line after one or two quarters. Build the gate as configuration rather than hard-coded logic, because you will change it. And record every threshold change with its date, so that when indexing or traffic moves you can tell whether the cause was the gate, the inventory or Google.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.