Discussion
  • Indexing
  • Internal Linking
  • Technical SEO

Internal linking for millions of templated pages: related-link modules, breadcrumbs or HTML sitemaps? Which moved indexation most?

Published 1 reply

Discussion starter: a programmatic SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    For getting leaf pages indexed, my default order is related-link modules first, hierarchical browse pages second, breadcrumbs third and flat HTML sitemaps last. Of the three you list, a related-links module is the only one that adds links into the long tail from pages Google already crawls often. Breadcrumbs point upwards, so they strengthen categories rather than helping deep pages get found. An HTML sitemap gives every URL a path, but at millions of URLs that path is long, unless the “sitemap” is really a browse hierarchy, at which point it is doing the job of category navigation.

    One condition matters more than the choice of module: links fix discovery, not rejection. If your unindexed pages sit in “Discovered - currently not indexed”, internal links are a main lever. If they sit in “Crawled - currently not indexed”, Google has fetched them and declined, and no link module is likely to change that on its own. No controlled public test that compares all three on one site exists that I can point to, so treat the order above as a default to test rather than a finding.

    First, find out which kind of “not indexed” you have

    Google’s Page indexing report documentation defines “Discovered - currently not indexed” as “The page was found by Google, but not crawled yet”, typically because Google expected crawling it to overload the site and rescheduled the crawl. “Crawled - currently not indexed” means “The page was crawled by Google but not indexed.”

    Google’s crawl budget guide, written for sites with a million or more pages changing weekly or 10,000 or more changing daily, adds the other half: “For Google Search, not every page that is crawled will necessarily be indexed. After crawling, each page must be evaluated, consolidated, and assessed to determine its suitability for the index.” It names perceived inventory, popularity and staleness among the factors behind crawl demand. Internal links affect discovery and, to a degree, how important a URL looks. They do not change what the page says.

    Split XML sitemaps by template, read the Page indexing report per sitemap, and check logs for whether Googlebot has ever requested the unindexed URLs
    ↓
    Mostly “Discovered - currently not indexed”? A discovery and crawl-priority problem: work on link paths, sitemaps and server response times
    ↓
    Mostly “Crawled - currently not indexed”? A value or duplication problem: fix the template’s content before adding links
    ↓
    Mostly duplicate statuses? A canonicalisation problem: consolidate before linking to either version
    ↓
    Only the first branch is mainly an internal linking problem

    The second branch deserves blunt treatment. If a template produces pages that differ only by a swapped place name or an empty result set, more links will not make Google want them. Pages generated at scale mainly to rank, rather than to help users, can also fall under the scaled content abuse section of Google’s spam policies.

    What each module does to the link graph

    Breadcrumbs: links upwards

    Every leaf links to its parents, so category and hub pages collect large numbers of internal links. That helps hubs get crawled and can help them rank. It adds no new path to any leaf, so its effect on leaf indexation is indirect: stronger hubs only help if the hubs link down well.

    HTML sitemaps: links downwards, complete

    Every URL can get a link, which removes orphans. At millions of URLs, a flat list becomes thousands of pages that are themselves deep and low-value, so Google has little reason to recrawl them often. Built as a hierarchy (region, then city, then category), it becomes browse navigation, which can work well.

    Related-link modules: links sideways

    On pages Google already crawls often, a module with eight links can put a very large number of pages within a few clicks. How well it works rises and falls with the selection logic: a module that links to what is already popular gives the tail nothing.

    XML sitemaps: not links, but part of the answer

    They tell Google a URL exists, and the crawl budget guide recommends lastmod for updated content. They do not give a URL the internal links that make it look important, so they complement linking rather than replace it.

    Hypothetical example:

    A property portal has 3 million listing and location pages. As a flat HTML sitemap at 500 links per page, it needs 6,000 sitemap pages, and if those are chained with “next” links, most of them sit hundreds or thousands of clicks from the homepage. As a hierarchy of 8 states, 600 regions and 15 property types, the 9,000 region-and-type pages each hold about 330 listings, so a listing sits four or five clicks down, plus pagination. Add a related-listings module with eight links per listing, chosen deterministically and spread across the whole inventory, and simple arithmetic says up to about 2.1 million pages sit seven clicks from any listing (8 to the power of 7). If the module instead picks the most-viewed listings in each region, a small head of pages collects most of the links and the rest get none.

    How related-link modules go wrong at scale

    • Popularity bias. Recommendation engines optimise for clicks, so they link to pages that are already crawled, indexed and visited.
    • Instability. Links that change on every request give Google a different graph on each crawl, and give you nothing stable to measure.
    • Uncrawlable markup. Google’s link best practices say “Generally, Google can only crawl your link if it’s an <a> HTML element (also known as anchor element) with an href attribute.” Modules built from click handlers or framework router attributes without an href fall outside that, and modules that load only after scrolling or clicking may never be seen.
    • Wasted slots. Links to noindexed, canonicalised, redirecting or expired URLs spend the module on pages that cannot be indexed.
    • Too many links. The same page says there is “no magical ideal number of links”, then adds: “However, if you think it’s too much, then it probably is.”

    A module built for indexation looks different from one built for engagement:

    • Plain <a href> links in the server-rendered HTML.
    • Deterministic selection for a given page (seeded by page ID, for example), changed on a schedule you control rather than on every request.
    • Relevance first: same category, nearby location, shared attributes. A related link no user would click is a sitemap entry in disguise.
    • A few slots reserved for pages that need discovery: new URLs, URLs with few inbound internal links, and URLs your logs show Googlebot has never requested. Rotate those slots weekly or monthly.
    • Targets limited to indexable, self-canonical URLs that return a 200.
    • Placement on the templates Googlebot requests most, which your logs will show.

    Where breadcrumbs and HTML sitemaps still earn their place

    Breadcrumbs belong on templated pages anyway. They show users and Google where a page sits, and they concentrate internal links on the hubs you want to rank. Just do not count them as an indexation tactic for leaves.

    An HTML sitemap is a reasonable safety net for Google’s baseline in its link guidance: “Every page you care about should have a link from at least one other page on your site.” At millions of URLs, though, the version that works is a browse directory built around real demand (“all suburbs we cover”, “all brands we stock”) rather than a dump of URLs. Built that way, it is category navigation under another name.

    How to test it on your own templates

    Indexation tests at this scale are more tractable than ranking tests. The outcome is close to binary per URL, and you have thousands of URLs per group. The design I would use:

    1. Randomise the targets, not the source pages. Split unindexed leaf pages into cohorts by a hash of their ID. The module may link to the treatment cohort and never to the control cohort, so the pages carrying the module are identical for both.
    2. Give each cohort its own XML sitemap. The Page indexing report can be filtered to a specific sitemap, so indexed and not-indexed counts per cohort come straight from Search Console. Submit both the same way, so sitemaps are not the difference.
    3. Measure in logs as well. Track days to first Googlebot request and requests per URL per week, by cohort. Logs move before the Page indexing report does.
    4. Set the duration by recrawl, not by calendar. The module does not exist for Google until the pages carrying it are recrawled. Start the clock when logs show most source pages fetched with the module in place, and run until the treatment cohort’s crawl activity levels off. If the source pages are hubs crawled daily, that may take a few weeks; if they are recrawled monthly, expect longer.
    5. Test breadcrumbs and HTML sitemaps separately. Both are sitewide and hard to randomise. Rolling them out to some sections while holding back comparable sections is the practical compromise, and the evidence it produces is weaker.
    6. Log confounders. A core update, server speed changes, sitemap changes and content releases can all move indexation in both cohorts at once.

    What would change the decision:

    • Treatment URLs get crawled but land in “Crawled - currently not indexed”. The module worked for discovery, and the template’s value is now the constraint. Improve or prune the pages before adding more links.
    • Neither cohort moves and Googlebot activity is flat. Check response times and server errors, because crawl capacity may be the limit.
    • Both cohorts improve together. Something sitewide changed, and the module is not proven.

    Where I would land

    If this were my site, I would split sitemaps by template and read the statuses before choosing a module. Where the backlog is discovery, I would make every template’s links crawlable HTML, keep breadcrumbs for structure, turn any flat HTML sitemap into a browse hierarchy, and put most of the effort into a deterministic related-links module with slots reserved for under-crawled pages, tested by cohort. Where the backlog is “Crawled - currently not indexed”, I would stop adding links and fix the template, because Google has already seen those pages and made its call, at least for now.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.