Discussion
  • Crawling
  • Technical SEO

Does splitting XML sitemaps by template and freshness actually change how Google crawls, or is it mostly a reporting convenience?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Mostly a reporting and operations tool, and Google’s own documentation frames it that way. You can submit several sitemaps and sitemap index files, and Google says this “may be useful if you want to track the search performance of each individual sitemap in Search Console”. Nothing in the documentation says a URL gets crawled sooner or more often because of which file it sits in. What can change crawling is which URLs you list and whether their lastmod values are accurate.

    The freshness half of the question needs a more careful answer than the template half. Splitting by freshness can help indirectly: it makes accurate lastmod easier to produce, and it puts changed URLs in a small file that changes often. Whether search engines fetch that kind of file more often is unconfirmed, including by Google. On a site with a few thousand URLs I would treat the whole exercise as reporting. On a large catalogue or marketplace with heavy daily change, the freshness part is worth testing in your own logs, and the test is cheap.

    “Reporting convenience” also undersells the template split. Seeing indexing by template can change decisions on a large site even when crawling does not change at all.

    What Google documents, and what it leaves out

    The sitemaps documentation is brief on the points that matter here. Submitting a sitemap is “merely a hint: it doesn’t guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site.” Google “ignores <priority> and <changefreq> values,” and it “uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.” A single sitemap is capped at 50MB uncompressed or 50,000 URLs, so above that size splitting is compulsory and the only decision is how to split.

    Two consequences follow. The format has no field for “crawl these first”, and Google does not document reading file names or file structure as a priority signal. And since Google retired its sitemap ping endpoint in 2023, a changed sitemap is picked up when Google next fetches it on its own schedule.

    A recent Google comment addresses the question almost directly. In a Reddit thread reported in April 2026, John Mueller answered why SEOs split sitemaps. He listed tracking groups of URLs separately (product detail pages versus category pages, for example), splitting by freshness with evergreen content in its own file, splitting early so you do not hit the 50,000 URL limit and have to scramble, and hreflang sitemaps, which take up a lot of space. On freshness, he suggested search engines might in theory check older sitemap files less often, then said he did not know whether that actually happens. The same write-up notes that some enterprise SEOs report that keeping files below the 50,000 URL maximum appears to help indexing. That is a field observation rather than a documented mechanism, and it is confounded: the teams splitting carefully may also be the ones cleaning their URL lists at the same time.

    Where a split could plausibly change crawling

    Mechanism Status What to expect How to check on your site
    Accurate lastmod on changed URLs Documented Changed URLs can be recrawled sooner once Google trusts the values. The split helps only if it makes accuracy easier to produce Hours from a content change to the next Googlebot fetch, from server logs
    A small, frequently changing “recent” file fetched more often Unconfirmed. Mueller called it possible in theory Possibly faster discovery of new URLs Googlebot requests per sitemap file, from logs
    Cleaner URL lists once per-template reporting exposes junk Indirect Fewer wasted hints once redirected, noindexed and non-canonical URLs are removed Page indexing report filtered by sitemap
    Failure isolation Operational A broken generator for one template stops taking every URL’s hints down with it Errors per file in the Sitemaps report, plus generator monitoring
    File order or naming as a priority signal No evidence Nothing Not worth testing
    lastmod trust judged per file Unknown. Google has not said whether it assesses accuracy per file, per host or site-wide Isolating a template with bad dates might protect the other files, or might not Fix the dates rather than relying on this

    The last row is a trap worth naming. If one template’s generator stamps every URL with the current date, moving it into its own file feels like damage control. Since Google has not documented the level at which it decides whether your lastmod values are trustworthy, the dependable fix is making them honest. In practice that means moving lastmod when the main content, structured data or meaningful links change, and not when a footer year, a view counter or the file’s generation time does.

    Why the reporting value is more than a convenience

    Search Console’s Page indexing report can be filtered to a single submitted sitemap, which turns one site-wide chart into one chart per template. On a large site, that can be the quickest way to find a template Google is declining to index. The Performance report cannot filter by sitemap, but its page filter accepts regular expressions, so templates with distinct URL patterns can be segmented there without any split. The Crawl stats report cannot be broken down by sitemap or template at all, so crawl behaviour by template still needs server logs joined to sitemap membership.

    Hypothetical example:

    A marketplace has 1.4 million URLs split into sitemaps for product pages, category pages, brand pages and suburb landing pages. Filtered by sitemap, the Page indexing report shows category pages almost fully indexed, product pages at a little over half, and suburb pages at around one in five, with most exclusions under “Crawled - currently not indexed”. The split did not change how Google crawled anything. It changed the decision: the suburb template gets consolidated or rebuilt, and the product template gets a closer look for thin and duplicate listings. A single sitemap would have shown one blended total and hidden which template was failing.

    A split that serves both purposes

    • One file, or one index of files, per template, named consistently so Search Console, logs and dashboards line up.
    • Only canonical, indexable URLs that return 200. Per-template reporting will show you where that rule is broken.
    • lastmod generated from real content changes, template by template.
    • Stable membership. URLs that hop between “fresh” and “archive” files make per-file trends hard to read, so express freshness through lastmod and keep each URL in its template file unless you are running the test below.
    • A log alert when Googlebot has not fetched a sitemap file for longer than its usual interval, and a check of the Sitemaps report after each release.
    • Earlier splits for hreflang-heavy sitemaps, since the annotations push files towards the 50MB cap well before they reach 50,000 URLs.

    Testing whether a freshness file changes crawling on your site

    This part can be tested properly because you control which URL goes where. Each day, randomly assign new or updated URLs to one of two arms. Arm A goes into a small “recent” sitemap. Arm B goes only into its normal template sitemap with the same accurate lastmod. Both arms need identical internal linking (new-listing modules, category placement, homepage feeds), otherwise you are testing links, not sitemaps.

    • Primary metric: hours from publication or update to the first verified Googlebot fetch, from server logs. Google moved its crawler IP range files in March 2026, so check that your verification script points at the current location.
    • Secondary metrics: how often Googlebot fetches each sitemap file, which tests Mueller’s theory directly, and time to indexed for a sample from each arm through URL Inspection, which has a daily quota.
    • Duration: set by volume, not the calendar. A site publishing a few hundred URLs a day has a couple of thousand per arm within two weeks, enough to compare medians and the slow tail. A site publishing ten a day would need months, and at that volume the question barely matters.
    • Confounders: release days, server errors or slow responses that reduce crawl capacity, and any change to internal link modules during the test.

    If arm A’s median time to first crawl is materially shorter, and the gap holds in a second period, keep a freshness file and keep it small. If the arms look the same, drop it and spend the effort on lastmod accuracy and on linking new URLs from pages Googlebot already visits often.

    How the answer shifts with scale and site type

    Site Does the split change crawling? What I would do
    Under about 10,000 URLs, modest change rate Very unlikely to matter. Google’s crawl budget guide is written for sites with over a million pages changing about weekly, over 10,000 pages changing daily, or many URLs stuck in “Discovered - currently not indexed” One sitemap is fine. Split only if templates cannot be separated by URL pattern in reports
    Large catalogue or marketplace with daily churn Possibly, through lastmod accuracy and perhaps file fetch frequency Template split for reporting, freshness through lastmod, and the arm test before building a freshness pipeline
    News publisher Google News sitemaps already build freshness into the format by listing only articles from the last two days Keep the news sitemap alongside the template sitemaps
    International site with hreflang in sitemaps File size is the binding constraint Split by market or language as well as template

    If this were my site, I would split by template for the diagnostics, generate lastmod from real content changes, and treat a freshness file as a hypothesis for the logs to confirm or reject. If the arms show no difference, the split is for you rather than for Googlebot. That is still worth having, as long as nobody presents it internally as a crawl budget fix.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.