Discussion starter: a programmatic SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Indexing
- Technical SEO
Has anyone built a model that predicts which templated pages Google will index? Which features turned out to matter?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Yes, it can be done, and it is one of the more useful things a programmatic team can build, but the public record of such models is thin and the results are specific to each site. The features that practitioners most often find predictive are not exotic: how unique the page’s content is compared with its nearest siblings, how much substantive content the template produces for that entity, how well the page is linked internally and how often it is crawled, and whether there is evidence of demand for the entity. Sitemap inclusion and markup rarely separate indexed from unindexed pages on their own. The biggest gains come less from the model’s accuracy than from using it as a gate before pages are published.
Frame the problem as two stages
“Will Google index this page?” hides two different decisions. The first is whether Google crawls the URL at all; the second is whether it keeps the page after crawling. Search Console reflects the split with “Discovered - currently not indexed” (known but not crawled) and “Crawled - currently not indexed” (fetched and not kept). The features that predict each stage differ. Crawling is driven heavily by discovery and linking; indexing after crawl is driven more by content value and duplication. Two models, or one model with the stage as an outcome, usually explain more than a single yes or no.
Getting labels you can trust
- URL Inspection API on a stratified sample gives the most direct label: index status, coverage state, last crawl time and Google-selected canonical. It has daily quotas per property, so sample by template and segment rather than trying to inspect everything.
- Page indexing report exports give example URLs for each reason, but they are samples and not complete.
- Search Console performance data can serve as a proxy (a page with impressions is indexed), but a page can be indexed and get no impressions, so this proxy mislabels low-demand pages.
- Server logs label the crawl stage: whether and how often verified Googlebot fetched each URL.
Features that tend to matter
Feature group Examples Why it plausibly matters Uniqueness Similarity to the most similar sibling page; share of text that is template boilerplate Pages that look like near-copies of many others are the classic “Crawled - currently not indexed” population Substantive content Number of listings, data points or reviews for the entity; length of entity-specific text Measures whether the page has anything to say beyond the template Internal linking Number of internal links in, click depth, links from pages that are themselves indexed and crawled often Drives discovery and crawl priority Crawl history Days since last Googlebot fetch, fetch frequency Strongly associated with the crawl stage; be careful of leakage when predicting indexing Demand Search volume for the entity term, internal search volume, impressions of similar pages Google appears to keep pages for which it sees a reason to serve them External signals Referring domains to the page or its parent Mostly relevant at higher levels of the hierarchy Technical consistency Canonical matches URL, status 200, no conflicting hreflang, render completeness Rules out pages that fail for technical reasons before quality is even considered Practitioner write-ups from programmatic sites tend to point the same way: uniqueness and substantive content carry the most weight for indexing after crawl, and internal linking and crawl frequency for the discovery stage. Treat that as a starting hypothesis to test on your own site, not a published finding.
Modelling choices and traps
- Start simple. A logistic regression or a small gradient-boosted tree model is enough. Use feature importance or SHAP values to see what drives predictions, and check that the drivers make sense.
- Avoid leakage. Features that are consequences of indexing, such as impressions or clicks, will make the model look brilliant and useless. Crawl frequency needs care for the same reason, since indexed pages get recrawled more.
- Watch for confounding by age. Older pages have more links, more crawl history and more chances to be indexed. Include page age explicitly, or train on pages of similar age.
- Validate on a later time period. Train on pages published in one quarter and test on the next. Google’s behaviour changes, and a model that only works on the past is a report, not a predictor.
- Retrain after core updates. Thresholds shift, and a model trained before an update can mislead after it.
Where the value really is:
The model’s most useful output is a list of pages you should not publish yet, or should improve before publishing. Every thin page you do not launch saves crawl activity and avoids adding to whatever site-level quality assessment Google makes. Prediction accuracy of 80% versus 85% matters much less than whether the prediction changes what goes live.
Hypothetical example:
A jobs aggregator has 400,000 location and job-title pages, of which about a third are indexed. A model trained on a stratified URL Inspection sample finds that pages with fewer than five live listings and pages whose listings overlap heavily with a neighbouring town’s page are rarely indexed, whatever their internal links. The team stops generating pages below the listing threshold, merges heavily overlapping neighbouring towns, and publishes new location pages only when the model predicts a good chance of indexing. The share of indexed pages among published pages is the metric that tells them whether the gate is working.
What to do with the predictions
- Gate publication: pages below a threshold stay unpublished or noindexed until they gain enough substance.
- Prioritise improvements: for pages just below the threshold, fix the features that the model shows are weakest.
- Adjust internal linking so pages most likely to be valuable are reachable in fewer clicks.
- Report the gap between predicted and actual indexing each month; a widening gap is an early warning that Google’s thresholds have shifted.
If you have not built one yet, start with a sample of a few thousand inspected URLs and a dozen features you already have in your database. You will learn more from the first simple model than from any benchmark someone else publishes, because the thresholds that matter are the ones Google applies to your site.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.