Discussion
  • Crawling
  • Site Architecture
  • Technical SEO

Now that rel=prev/next is ignored, which pagination setup gave you the best deep-item discovery: view-all, numbered pages, or load-more with crawlable links?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    For deep-item discovery, numbered pages with plain links are still the most dependable of the three. “Load more with crawlable links” is not really a separate option for a crawler: done properly, it is a next-page chain with a nicer interface for people, and it inherits the weaknesses of a next-only chain. View-all is the only structurally different choice, and it works well only while the full list stays small and light.

    The bigger point is that pagination is a weak discovery path at any scale. On a large catalogue, whether an item on page 60 gets crawled depends more on how many clicks it sits from page 1, and whether it has any other way in (subcategories, indexable facets, related-item links, sitemaps), than on which of the three patterns you picked. So the useful decision is: one long page or a series, and then how shallow can the series be made.

    What Google has documented since rel=prev/next went

    Google’s pagination and incremental loading guidance is short and fairly specific:

    • Link from each page to the following page with <a href> tags so Googlebot can find later pages.
    • “Don’t use the first page of a paginated sequence as the canonical page. Instead, give each page its own canonical URL.”
    • Google’s crawlers “don’t ‘click’ buttons and generally don’t trigger JavaScript functions that require user actions to update the current page contents.”
    • Don’t put page numbers in URL fragments, because Google ignores everything after the #.
    • Consider linking every page in the series back to page 1.

    The lazy-loading documentation covers the infinite scroll version: give each chunk “its own persistent, unique URL”, keep the content at that URL the same each time it loads, link sequentially to those URLs, and update the displayed URL with the History API when a new chunk becomes the main visible content. It also states plainly that “Google Search does not interact with your page.”

    View-all is not addressed in the current pagination page. When Google retired rel=prev/next in 2019, it said users tend to prefer single-page content and suggested aiming for that where possible, while noting multi-part content is fine for Search. That is a user-experience preference. It does not tell you what happens when the single page holds 4,000 products.

    Where each pattern breaks

    Pattern How deep items get found Where it breaks What to check
    View-all Every item is linked from one URL, so each is one hop from the list. Large lists: heavy HTML, slow rendering, poor mobile experience. Google documents a 2MB fetch limit for HTML, and bytes beyond it are not indexed, so items near the end of a huge list can simply be cut off. Uncompressed HTML size of the view-all response; whether the last items appear in URL Inspection’s rendered HTML.
    Numbered pages Each item sits on one component page, so it depends on that page being crawled. Next-only links create long chains. Sort orders that reshuffle items between crawls. Page 2 onwards canonicalised to page 1 or noindexed. Googlebot requests by page number in server logs.
    Load more with crawlable links The button is a real <a href="?page=2"> that JavaScript intercepts; crawlers follow the href. The “link” is a <button> or href="#". ?page=2 returns a JSON fragment, or page 1, when requested directly. Only a next link exists, so depth grows one hop per page. Request ?page=5 in a clean session with JavaScript off and confirm it returns the fifth chunk with onward links.
    Infinite scroll with no paginated URLs Only the first chunk is reliably seen. Everything beyond the first batch has no crawlable path from the list. Compare the item count in the rendered HTML with the category total.

    Why click depth decides deep-item discovery

    A next-only series works like a library where each shelf carries a note saying where the next shelf is. The librarian can reach shelf 80, but only by following 79 notes, and each note is found only when the previous shelf is visited again. Numbered links with jumps put a floor plan at the entrance.

    The technical version is less literal. Googlebot does not walk a series in one session. It queues discovered URLs and schedules them by perceived importance and expected change. Page 1 of a category gets links from navigation and other categories. Page 40 gets a link from page 39 and not much else, so it is crawled rarely, and an item linked only from page 40 is discovered late and prioritised low. A sitemap fixes the discovery part, but it carries little signal about importance, which is how deep items end up sitting in “Discovered - currently not indexed” even though Google knows they exist.

    Hypothetical example:

    A category holds 5,000 items at 24 per page, so 209 pages. With next-only links (which is what a plain load-more gives a crawler), the last page is 208 hops from page 1. If every page instead links to the five pages either side of it plus every tenth page (10, 20, 30 and so on up to 200) and the last page, any page in the series is at most two hops from page 1. That costs about 30 pagination links per page, which is a reasonable trade for a category that size.

    Sort order matters as much as the link pattern. A newest-first sort puts new items on page 1, which is excellent for discovering new stock, but it pushes older items deeper every day and changes what each ?page=N URL contains between crawls. A best-seller sort concentrates the shallow pages on items that already have the most signals. Decide which items you need discovered, then check whether the default sort serves that goal.

    When view-all is the better choice

    View-all suits categories with dozens to a low few hundred items, where the full list renders quickly and the HTML stays well under the fetch limit. In that case it gives every item a one-hop link from a strong URL, and users can scan and use find-in-page. If you keep both a view-all and component pages, canonicalising the components to the view-all is coherent, because the view-all contains everything on them. That is different from canonicalising page 2 to page 1, which points at a page that does not contain page 2’s items.

    It stops being the better choice when the list is large, when items carry heavy markup (image galleries, variant swatches, inline reviews), or when the page relies on JavaScript to render items as the user scrolls. At that point you are trading a depth problem for a size and rendering problem, and the size problem fails silently.

    What “load more with crawlable links” needs to get right

    This is the pattern most often implemented badly, because it looks correct in a browser. A working version has these properties:

    • The control is an anchor with a real href to ?page=2, enhanced by JavaScript, not a button with a click handler.
    • Each ?page=N URL, requested directly, server-renders its own chunk with a link to the next and previous chunk.
    • Each chunk URL is self-canonical and indexable, and links back to page 1.
    • The address bar updates with the History API as chunks load, so a shared or refreshed URL returns the same content.
    • A compact numbered navigation (the same jump pattern as above) appears somewhere on each chunk URL. Without it, load-more is a next-only chain.

    That last point is where load-more most often loses to numbered pages for deep items. The user sees one continuous list, but a crawler sees a long corridor.

    How to measure deep-item discovery on your own site

    Impressions are a lagging and noisy measure here. I would measure discovery and crawling directly, bucketed by how deep the item sits:

    1. From server logs, count verified Googlebot requests to paginated URLs by page number over 30 days. A sharp fall after page 3 to 5 means items that live only deeper are relying on other paths.
    2. Assign each item to the page number where it first appears in its main category, then measure the share crawled in the last 30 days and the share indexed (using the Page indexing report or a URL Inspection API sample) per bucket.
    3. For a sample of deep items, check the “Referring page” in URL Inspection to see how Google actually found them. If the answer is mostly sitemaps, pagination is not doing the discovery work.
    4. For new items, record the time from publish to first Googlebot request, split by bucket.
    5. Look for “Discovered - currently not indexed” clustering on items from deep buckets rather than on a particular template or brand.

    What to test, and which result would change the decision

    A reasonable test: pick two groups of categories of similar size and crawl rate. In the test group, add numbered links with jumps (or add numbered navigation to existing load-more URLs). Leave the control group unchanged. The primary metric is the share of items on pages 10 and beyond that receive a Googlebot request within 14 days. Secondary metrics are indexed share for those items, their impressions, and the proportion of Googlebot requests spent on pagination rather than items. Run it for four to eight weeks, avoid changing sitemaps or internal linking elsewhere at the same time, and annotate any core update that lands in the window.

    Two results should change the plan. If deep items are already crawled promptly through sitemaps, subcategories and related-item links, the bucket gap will be small and pagination changes are not worth an engineering sprint. If pagination crawling rises but deep items still are not indexed, the problem is item quality or duplication, not discovery, and more links will not fix it.

    Mistakes that undo any of the three:

    Canonicalising page 2 onwards to page 1. Blocking ?page= in robots.txt. Leaving paginated pages noindexed for the long term (Google’s John Mueller has said long-term noindexed pages tend to be treated as nofollow as well, so their links stop helping). Letting filter and sort parameters multiply the series; Google’s guidance is to keep those variations out of the index with noindex. Leaving rel=prev/next in place is harmless, but it does nothing for Google.

    If this were my catalogue, I would default to numbered pages with a jump pattern, keep load-more as the user interface on top of real paginated URLs where the design calls for it, and reserve view-all for categories small enough to render quickly. Then I would let the log buckets decide whether the next piece of work belongs in pagination at all, or in the subcategory and related-item links that usually carry deep items better.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.