Discussion
  • Crawling
  • Technical SEO

Crawlers flag thousands of orphan URLs that still earn clicks through XML sitemaps. Would you link them, prune them, or leave a working system alone?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    I would do all three, to different segments, after confirming the orphans are real. The URLs earning clicks and still serving a current purpose should be linked, because they are doing well despite missing a signal Google documents, and their only discovery path is a sitemap that can break. The ones with no clicks, impressions, links or conversions over a full data window are pruning candidates. And a middle group can be left alone deliberately, as long as “working” is defined by more than a click count.

    The deciding factor is not the orphan flag. It is why each URL became orphaned, what it earns, and what the site loses if it disappears or keeps sitting outside the architecture.

    First, check whether they are really orphans

    An orphan report is only as good as the crawl that produced it. Before anyone opens a ticket, I would rule out the common false positives:

    • Crawl limits: depth caps, URL caps or paginated archives the crawler stopped following. Old blog posts and deep products are often linked on page 40 of an archive, not unlinked.
    • Rendering: links injected by JavaScript that the crawler did not render. If those links are real <a href> elements in the rendered HTML, Google can likely follow them, so the fix is the crawl setting, not the site.
    • Excluded areas: faceted navigation, parameters, subdomains or country folders deliberately left out of the crawl.
    • Links that are not crawlable links: internal search results, onclick handlers, forms or buttons without an href. These look like links to users but are genuine orphans for search engines.
    • Canonicalised or redirected URLs: a sitemap listing URLs that canonicalise or redirect elsewhere shows up as “orphans”, when the real problem is sitemap hygiene.

    Rerun a targeted crawl on a sample with rendering on and limits lifted. If a large share stops being orphaned, the finding was a crawl configuration issue.

    Why a sitemap-only URL can rank, and why it is still fragile

    Think of the sitemap as the room list handed to a building inspector and internal links as the corridors and signs. The inspector can visit every room on the list. But visitors can only reach rooms connected by corridors, and the signs tell everyone which rooms matter. An orphan page is a room on the list with no door from the hallway.

    In search terms, the stages separate like this:

    Discovery: the sitemap supplies the URL
    ↓
    Crawling: Google schedules and fetches it
    ↓
    Indexing: content quality and canonical signals decide inclusion
    ↓
    Ranking: relevance, external links, history and internal links all feed in

    The sitemap only solves the first stage. Google’s documentation says a sitemap “helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed”. The same page notes that on large sites it is harder to ensure every page is linked from at least one other page, which is exactly why sitemaps exist as a backstop.

    Google’s link guidance is more direct: “Google uses links as a signal when determining the relevancy of pages and to find new pages to crawl,” and “Every page you care about should have a link from at least one other page on your site”. So an orphan that earns clicks is ranking without one documented input. That is evidence it has other strong signals, such as external links, history or a clear match with a niche query. It is not evidence that internal links would add nothing.

    Important distinction:

    Sitemaps do not earn clicks. They got the URL discovered. The clicks come from rankings, and those depend on everything else about the page. Taking a URL out of the sitemap is also not a removal signal: Google already knows the URL and can keep crawling it, usually less often.

    The fragility is real but slower than people fear. If a sitemap generator breaks, known URLs do not vanish overnight. What you lose is discovery for new orphan-style pages, a fresh lastmod hint for updated ones, and any internal path for users who land elsewhere on the site. The bigger risk is often commercial: pages nobody on the team can navigate to tend to go stale, carry wrong prices or policies, and get missed in migrations.

    Segment before deciding

    Join these sources at URL level: the orphan list, the sitemap, Search Console clicks and impressions (the full 16 months), conversions or revenue by landing page, Googlebot hits from logs, referring domains, and a sample of indexing status through URL Inspection. The URL Inspection API has a daily quota per property, so inspect a stratified sample rather than every URL.

    Segment Typical cause Lean towards Evidence to check first
    Clicks, conversions, current content Dropped from navigation in a redesign or category restructure Link from the most relevant hub, category or related-content module Whether a linked page already covers the same intent
    Clicks, but outdated or near-duplicate of a linked page Old versions, seasonal pages, CMS migration leftovers Consolidate with a 301 to the linked equivalent, on a test batch first Query overlap between the two pages, backlinks to the orphan
    Impressions but few clicks Long-tail pages generated without link modules Link a test group, then compare against an unlinked group Average position trend and crawl frequency
    No clicks, impressions, links or conversions in 16 months Abandoned content, thin programmatic pages Prune: 404 or 410, or noindex if the page must stay for users; remove from the sitemap Referring domains and internal stakeholder use
    Intentional orphans Paid landing pages, email-only offers, thank-you pages Remove from the sitemap; noindex if they should not rank Whether paid campaigns still point at them
    Discontinued products still earning clicks Product removed from category but page kept live Keep with alternatives linked, or redirect to the closest replacement Conversion path from the page and returning demand

    Scale changes the weighting. Google’s crawl budget guide is aimed at sites with over a million unique pages changing about weekly, or 10,000 or more changing daily. Below that, pruning is a quality and maintenance decision, not a crawl-efficiency one. On a large marketplace, thousands of dead orphans in the sitemap make it a weaker list of what matters and can absorb crawling you would rather see spent on fresh inventory.

    Hypothetical example:

    A retailer with 40,000 indexable URLs has 6,000 confirmed orphans. About 1,100 earned clicks in the last 16 months, and 250 of those drive most of the orphan revenue: they are current products dropped from filtered category pages after a navigation change. Another 400 are discontinued products with steady clicks. Around 3,800 have no clicks, impressions or links. The decision is not “link 6,000 URLs”. It is: restore the 250 to navigation, fix the discontinued pages so they lead somewhere useful, test linking on the long-tail group, and retire the 3,800 in batches while watching for anything that was quietly earning.

    When leaving them alone is the right call

    “Don’t fix a working system” is a legitimate position for a subset. It holds when:

    • The pages earn steady clicks, the content is still accurate, and linking them would mean forcing unrelated items into navigation that serves core commercial paths.
    • Engineering time would come from something with a clearer return, such as template-level fixes affecting far more traffic.
    • The sitemap generation is reliable and monitored, with an alert if URL counts drop suddenly.

    It stops holding when orphans include your revenue pages, when nobody owns their accuracy, or when the orphan count keeps growing because the publishing process creates pages without linking them. In that last case the orphans are a symptom, and the fix is a rule in the CMS or template (every new page must attach to a parent hub), not a one-off cleanup.

    How to test whether linking helps

    Internal links are one of the few ranking inputs you can test cleanly on your own site, because you control the change and can hold back a control group.

    1. Hypothesis: adding contextual internal links to orphans with impressions will increase their impressions and clicks.
    2. Groups: split a homogeneous segment (same template, similar demand) randomly into test and control, a few hundred URLs each if you have them.
    3. Change: add links from relevant, crawled pages with descriptive anchor text. Change nothing else on either group.
    4. Measure: clicks and impressions per URL from Search Console, and Googlebot hits from logs, for six to eight weeks. The control group absorbs seasonality and core update noise.
    5. Decision: if the test group outperforms control beyond normal week-to-week variation, roll out to similar segments. If there is no difference, linking becomes a user-experience and maintenance decision rather than a traffic one, and priority drops.

    Pruning risks worth planning for

    Watch for this:

    Do not delete orphans that have referring domains without deciding where that equity should go; a 301 to a close equivalent keeps it, a 410 does not. Do not leave pruned URLs in the sitemap, and do not list noindexed URLs there either, because the sitemap then contradicts the page. Retire in batches so you can see whether any group was quietly earning traffic or conversions the data join missed.

    If this were my site, I would verify the orphan list, protect and link the small set of orphans that earn revenue, test linking on the long-tail group before touching the rest, and retire the pages that fail on clicks, impressions, links and conversions together. Then I would fix the publishing rule that created them, so the next crawl does not find another few thousand.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.