Discussion starter: a programmatic SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Technical SEO
How are you auditing programmatic sites to show each page has distinct value, given Google’s scaled content abuse policy?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
The short answer: audit the data each page carries, not the template’s prose, and do it page by page. A programmatic page has distinct value when it holds information a user needs that no sibling page holds, and when there is real demand for that specific combination. Everything else is presentation.
The audit’s job is to find the pages that fail that test and take them out of the index, or stop generating them. You cannot certify a site as compliant to Google. What you can do is make sure indexing is gated on page-level evidence, and then watch whether Google’s own indexing decisions agree with your scores.
What the policy actually targets
Google’s spam policies define scaled content abuse as “when many pages are generated for the primary purpose of manipulating search rankings and not helping users.” The examples include “Scraping feeds, search results, or other content to generate many pages (including through automated transformations like synonymizing, translating, or other obfuscation techniques), where little value is provided to users” and “Stitching or combining content from different web pages without adding value.”
Two things follow for programmatic sites. First, automation is not the problem. Google’s guidance on AI-generated content makes the same point: the test is value to users, not how the content was produced. A real-estate site with a page per suburb, each built from that suburb’s own sales data, is programmatic and legitimate. Second, the same page describes doorway abuse as pages “created to rank for specific, similar search queries” that “lead users to intermediate pages that aren’t as useful as the final destination.” City-by-service pages that funnel everyone to the same form are the most common programmatic pattern that meets this definition.
Google’s quality rater guidelines were updated in January 2025 to point raters the same way. According to Search Engine Land’s coverage, raters are told to give the Lowest rating when nearly all of a page’s main content is copied, paraphrased, auto-generated or AI-generated with “little to no effort, little to no originality, and little to no added value” for visitors. Raters do not rank pages. But the guidelines show what Google’s systems are meant to reward.
The unit of audit is the data payload, not the template
Think of a programmatic site as a set of identical shelves (the template) holding different stock (the data). A shopper does not care that the shelves match. They care whether the shelf in front of them holds something the next shelf does not. If every shelf holds the same tin with a different label, you have one product and ten thousand labels.
In practice, most of what a template renders is identical across pages: navigation, intro copy, FAQs, disclaimers and generated paragraphs describing the data. The value comes from the fields that change: listings, prices, availability, measurements, reviews, schedules, local rules, comparisons. Your audit therefore needs to measure those fields directly, ideally from the database rather than from a crawl, because the database tells you which fields are populated, which are defaults and which are copied from a parent record.
Watch for this:
Using an LLM to rewrite each page’s intro so that similarity scores drop does not create distinct value. It creates the “automated transformations like synonymizing” that the policy names. If your similarity score improves only because the prose changed while the data stayed thin, the audit has measured the wrong thing.
The page-level tests I would run
Test What it shows How to measure it Fail signal Data density Whether the page carries its own information Count page-specific, non-default fields and records (for example the number of listings, reviews or data points) from the database Below a threshold you set per template, such as zero or one listing Near-duplicate distance Whether the page differs from its closest sibling in a meaningful way Strip boilerplate, then compare the remaining main content with shingle or MinHash similarity, or embeddings; crawler near-duplicate reports are a starting point The nearest sibling differs mainly in entity names Swap test Whether the content is really about this entity On a manual sample, swap the city or product name for a sibling’s and reread the page The page still reads as accurate after the swap Demand Whether anyone searches for this specific combination Search Console impressions per URL over 3 to 12 months (via the API or bulk export, not the capped UI export), plus keyword data for new pages No impressions and no evidence of demand User outcome Whether visitors do something useful there Engagement, conversions and onward clicks in analytics, and internal search or navigation use Visitors bounce straight to a parent or sibling page Google’s response How Google already treats the cohort Page indexing report by cohort, crawl frequency from server logs High share of “Crawled - currently not indexed” or “Discovered - currently not indexed” The last row is the one practitioners underuse. Split your XML sitemaps by template and by data-density band, for example one sitemap for pages with 20+ listings and another for pages with one to four. The Page indexing report filtered by sitemap then shows Google’s indexing rate for each band. If the thin band sits mostly in “Crawled - currently not indexed” while the rich band is indexed, Google is telling you where it draws the line. For page-level checks, the URL Inspection API has daily quotas, so sample rather than trying to inspect everything.
Turning scores into index decisions
Scores are only useful if they drive a rule the site enforces automatically. I would combine the data-density and demand tests into a simple matrix, then use the other tests to handle edge cases:
Demand present No measurable demand Rich, distinct data Index. Improve internal links and the page experience. Keep for users who navigate there. Index selectively and stop creating more pages of this type. Thin or copied data Consolidate into a parent page (region, category) or enrich before indexing. Demand without data means you are not ready for this page yet. Noindex, or do not generate. Remove with a 404 or 410 if the page has no user purpose. Then put the rule in code. The pattern that holds up best on large programmatic sites is to treat indexability as a computed property: a page becomes indexable when its data crosses a threshold, and drops back to noindex (or out of the sitemap) when it falls below. That turns the audit from a one-off clean-up into an ongoing control, and it is the best answer to the question of how you know each page has value. Every indexable page has passed a documented rule.
Hypothetical example:
A trades directory has 180,000 “service in suburb” pages. The database shows 22,000 with five or more real businesses listed, 61,000 with one to four, and 97,000 with none, which show only nearby results and generated copy. The sitemap split shows most of the zero-listing band excluded from the index already. The rule becomes: index at five or more listings; one to four listings rolls up into a district page that is indexable; zero listings is noindexed and removed from sitemaps, and new pages are only generated when a business joins. The site ends up with far fewer indexable URLs, each of which can show why it exists.
What does not prove distinct value
- Unique titles and meta descriptions. These show the template can insert variables, not that the page is useful.
- Word count. Generated paragraphs that describe the data in prose (“Brisbane has 14 plumbers with an average rating of 4.6”) add length, not information.
- Generic FAQs and “about this area” sections. If the same answers would work on every sibling page, they are boilerplate with a variable in them.
- Traffic to a handful of pages. A few winners do not justify tens of thousands of zero-demand siblings. Judge by cohort.
Google’s helpful content questions give you the frame for writing this up: “Does the content provide substantial value when compared to other pages in search results?” and “Are you producing lots of content on many topics in hopes that some of it might perform well in search results?” If an honest answer to the second question is yes for a cohort, the audit should say so.
How scale changes the audit
On a few thousand pages you can review a large sample by hand, and the swap test does much of the work. From tens of thousands of pages upwards, the audit has to run on the database and on Search Console exports, with manual review kept for a stratified sample from each band so you can check the scoring. At millions of URLs, crawl budget joins the list of concerns: Google’s crawl budget guide is written for exactly this scale, and thin cohorts that attract heavy crawling are worth finding in the logs.
Business model matters too. A marketplace whose thin pages still convert a few paid enquiries may choose to keep them indexed while it builds supply. That is a defensible trade-off if the pages show real inventory. It is much harder to defend if the pages show “no results found” wrapped in generated copy.
Testing a pruning decision before rolling it out
- Hypothesis: noindexing or consolidating the thin band will not reduce total organic clicks and may improve crawling and indexing of the rich band.
- Test group: one template or a group of regions where you apply the rule. Control: comparable regions left unchanged.
- Primary metric: organic clicks to the whole cohort, including parent pages that absorb consolidated URLs. Secondary: indexing rate of the rich band, crawl frequency in logs, conversions.
- Duration: at least several weeks, avoiding seasonal peaks. Record the change and any core update (Google ran core updates in March and May 2026) in Search Console annotations.
- Decision: if total clicks hold or rise, extend the rule. If the pruned pages were quietly bringing in conversions, raise the threshold more slowly or enrich instead of removing.
Be careful about reading improvements as proof that Google was suppressing the site for scaled content. Recovery after pruning may come from better crawling, from consolidated signals or from an unrelated update. The test tells you whether the rule is safe, not why rankings moved.
The practical position
If I were auditing a programmatic site against this policy, I would start from the database, not the crawl: data density per URL, split into bands, with separate sitemaps per band so Google’s indexing decisions are visible. I would add demand and near-duplicate scores, hand-check a sample from each band with the swap test, and write the result as an indexing rule the platform enforces. The deliverable is not a report saying the pages are fine. It is a threshold, the evidence behind it, and a site that no longer asks Google to index pages that cannot show why they exist.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.