Discussion
  • Crawling
  • Enterprise SEO

On sites with millions of URLs, when do you act on a sampled crawl, and when do you insist on a full crawl first?

Published 1 reply

Discussion starter: an enterprise SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    The short answer: act on a sample when the question is about a rate or a template (“what share of product pages render without the description?”, “does the listing template emit the wrong canonical?”), and insist on full coverage when the question is about completeness or relationships (“which URLs need a redirect?”, “what is orphaned?”, “how deep is this section?”). The first kind of question has a statistical answer. The second kind does not, because every missed URL is a missed answer.

    The qualification most teams skip: a crawl that stopped at 500,000 URLs is not a sample. It is a truncated crawl, and it is biased towards whatever is closest to the start URL. A real sample needs a sampling frame (a complete list of URLs from the database, sitemaps and logs) and a random or stratified draw from it. On a site with millions of URLs, the decision usually depends less on crawler capacity than on whether you have that inventory and whether the fix is applied per template or per URL.

    Why a capped crawl gives you the wrong kind of partial answer

    A spider discovers URLs by following links, usually breadth first. If you cap it, you get the shallow, well-linked part of the site in full and the deep part not at all. On a marketplace, that means categories and the first few pages of listings are over-represented, while deep pagination, long-tail products, old listings and most facet combinations are missing. Any rate you calculate from that crawl (“2% of URLs have duplicate titles”) describes the shallow site, not the whole site.

    Important distinction:

    A truncated crawl answers “what does the top of the link graph look like?”. A random sample from a full inventory answers “what is true of the site as a whole, within a margin of error?”. A full crawl answers “what is true of each URL we could reach by links?”. These are three different questions, and teams often use the first to answer the second.

    The fix is cheap compared with a full crawl. Build the URL universe from sources that do not depend on crawling: a database or CMS export, the XML sitemaps, a few weeks of server logs (what Googlebot actually requests), and the URLs Search Console reports. Deduplicate, tag each URL with its template and section, then draw your sample from that list and crawl it in list mode. Now the sample can be random, and the URLs that the spider would never have reached have the same chance of selection as the homepage neighbours.

    What a sample can and cannot tell you

    Sampling works well for estimating proportions. The sample size you need depends on the precision you want, not on the size of the site. For a proportion near 50%, about 385 randomly chosen URLs per segment gives roughly plus or minus 5 percentage points at 95% confidence, and about 2,400 gets you to plus or minus 2 points. That holds whether the template has 50,000 URLs or 5 million. This is why a stratified sample of a few thousand URLs, split by template, is usually enough to size a problem and prioritise it.

    Sampling works badly for rare defects. Suppose a data condition breaks the canonical on 0.1% of product pages. A random sample of 2,000 URLs would expect to catch two of them, and there is roughly a 13% chance of catching none. The useful rule of thumb here is the “rule of three”: if you find zero defects in n random URLs, the 95% upper bound on the true rate is about 3/n. A clean sample of 3,000 URLs tells you the rate is probably below 0.1%. On a 5 million URL site, 0.1% is still 5,000 URLs.

    Watch for this:

    A clean sample never proves absence. It puts a ceiling on the rate. Whether that ceiling is acceptable depends on which URLs could be hiding under it. Five thousand broken pagination pages may not matter. Five thousand broken top-selling product pages would.

    Sampling also breaks anything that is a property of the link graph rather than of a single page. Crawl depth, internal link counts, orphan detection, internal link equity models, redirect chains and hreflang reciprocity all depend on relationships between URLs. You cannot estimate how many URLs are orphaned from a crawl, because orphans are by definition the URLs the crawl cannot reach. You find them by comparing a full crawl against a full inventory.

    There is a middle path for relationship checks: sample, then expand one hop. Crawl the sampled URLs and also fetch their canonical targets, redirect destinations and hreflang alternates. That lets you check whether sampled hreflang tags are reciprocal, or whether sampled canonicals point to 200 pages, without crawling everything. It still cannot give you depth or orphan counts.

    A decision table for common large-site questions

    Question you are answering Sample enough? Why What I would use
    Does this template render, canonicalise and title pages correctly? Yes Template defects are usually systematic; a few hundred URLs per template will show them Stratified sample by template, rendered crawl
    What share of listings are thin, duplicate or out of stock? Yes It is a rate estimate used for sizing and prioritisation Random sample per segment, plus a database query if the field exists
    Are there rare data-driven defects (odd characters, missing fields, bad canonicals on edge cases)? Usually no Low-rate defects slip through samples Query the source data directly, or a full crawl of the affected template
    Redirect mapping for a migration No Every URL with traffic, links or index presence needs a destination Full inventory from database, logs, backlinks and Search Console, then a full list-mode crawl
    Orphans, crawl depth, internal link distribution No Graph properties need the whole graph Full crawl compared against full inventory
    hreflang health Partly Rates can be sampled; reciprocity needs both sides Sample plus one hop; full crawl before a market launch
    Legal or compliance removals No A single miss can matter Database query and verification of every URL, not a crawl estimate
    Pre-release QA of a template change Yes, then monitor You need fast go or no-go on staging Sample on staging, then a full crawl of the changed template after release

    Three questions that decide it

    Is the fix applied per template or per URL?
    ↓
    Is the property local to a page, or does it depend on links between pages?
    ↓
    What does one missed URL cost?
    ↓
    Template + local property + low cost of a miss: act on a stratified sample. Anything else: get full coverage of the affected segment.

    The unit of action matters most. If the fix is a template change, the sample only has to establish that the template is wrong and roughly how many pages it touches. Engineering will fix all of them at once. If the fix is a per-URL list (redirects, removals, canonical overrides), the sample cannot produce the list, so a full crawl or a full data query is unavoidable.

    Reversibility matters next. Changing a template’s title logic based on a sample is easy to undo. Noindexing or redirecting a large URL set based on a sample-derived rule is harder to undo, and it can take weeks for Google to recrawl and reprocess. For irreversible or slow-to-reverse actions, I would at least run a full crawl of the segment the rule will touch before shipping.

    Hypothetical example:

    A classifieds site has about 6 million URLs: 40,000 category pages, 1.2 million listings and the rest facet and parameter URLs. A crawl capped at 1 million URLs from the homepage reports healthy canonicals. A stratified sample of 1,000 URLs from each of six segments, drawn from a database export plus 30 days of logs, shows that listings older than 12 months (never reached by the capped crawl) canonicalise to a retired URL pattern. The sample is enough to open a ticket and size the problem. The fix, however, involves updating canonicals on a specific set of listings, so the team queries the database for every affected ID rather than crawling to find them.

    Where full crawls fall short too

    A “full crawl” of a site with millions of URLs is not a clean snapshot. At 10 URLs per second, 5 million URLs takes close to six days, and listings expire, prices change and pages get added while the crawl runs. It also only reaches what is linked, so it can miss URLs that Google still knows about from old sitemaps, backlinks or history. And a crawl shows what the site exposes, not what Google does with it.

    That is why logs and Search Console belong in the process. Logs show which URLs Googlebot requests and how often, which is the only direct view of crawl allocation. Google’s crawl budget guide is aimed at exactly this scale (1 million+ unique pages changing about weekly, or 10,000+ changing daily), and logs are how you check whether its recommendations apply to you.

    Search Console, on the other hand, is sampled by design at this scale. Google’s help page says the Page indexing report’s example list is limited to 1,000 rows and is not guaranteed to show every URL in a status, even when fewer than 1,000 are affected. The URL Inspection API allows 2,000 queries per day per property, so inspecting 1 million URLs through one property would take about 500 days. Practitioners work around this with sitemap filters and subfolder properties, but index status at URL level is effectively a sample on any site this size. Treat it as a stratified sample and draw it deliberately, rather than trusting whatever examples the report happens to show.

    How I would validate a sample before acting on it

    1. Check the frame, not just the sample. Compare the inventory against logs and sitemaps. If Googlebot requests URL patterns that are missing from your inventory, the frame is incomplete and the sample inherits that blind spot.
    2. Stratify by template and by anything that changes behaviour: age, stock status, indexability rules, locale. A single random sample on a site where 80% of URLs are facets will tell you mostly about facets.
    3. Draw two independent samples for the metric that drives the decision. If they disagree by more than the margin of error, your strata are hiding something.
    4. Where a ground truth exists (a database field, a render flag), compare the sample estimate against it once. That tells you whether the crawler configuration itself is distorting results, for example through timeouts or blocked resources.
    5. After the fix ships, run a full crawl of the affected segment only. That is far cheaper than a full-site crawl and closes the rare-defect gap the sample left open.

    The practical position

    If this were my site, I would keep a continuously updated URL inventory and a stratified sampling routine as the default, because that answers most prioritisation questions within hours instead of days. I would insist on full coverage for four situations: migrations, anything that produces a per-URL action list, link graph questions (depth, orphans, internal links), and any segment where a single miss carries commercial or legal risk. I would also never accept a capped crawl’s percentages as site-wide numbers. Before acting, ask whether the evidence came from a real sample of the whole site or from the top of the link graph. Answering that question prevents more bad decisions than choosing between sampled and full crawls.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.