Pagination and AI Search: Keeping Collections Retrievable

Pagination is any scheme that splits one logical collection — a product category, a blog archive, a directory of listings — across multiple URLs or a single scrolling interface. The retrievability question is narrower and more concrete: can a crawler that never scrolls, never clicks, and never executes JavaScript reach the 47th item in your list, and does that item end up at a URL stable enough to cite? For most sites built in the last five years, the answer is no by default, not because pagination is inherently broken but because the two dominant UI patterns — infinite scroll and JavaScript-driven "load more" — were designed for humans with mouse wheels, not for bots reading raw HTML.

Why rel="next"/rel="prev" stopped being the answer

Direct answer: For most of the 2010s, the standard advice was to mark up paginated series with rel="next" and rel="prev" link tags in the page head section, a convention Google itself introduced in a 2011 Search Central blog post. That post now carries an editor's note at the top stating the markup is no longer an indexing signal, and Google confirmed in March 2019 that it had quietly stopped using rel=next/prev — a change Search Engine Land's 2025 pagination guide traces back to that announcement and the resulting uncertainty about what replaced it.

Nothing replaced it, exactly. Google's position, reflected in its current documentation, is that it treats each page in a paginated series as a normal page and relies on ordinary crawling and link-discovery signals — not a special pagination markup — to understand the series. Leaving old rel=next/prev tags in your code doesn't hurt anything (Bing still reads them), but adding them today does nothing for Google, and betting your collection's crawlability on them is a mistake.

What Google's current guidance actually says

Direct answer: Google's live reference for this is "Pagination Best Practices for Google" in Search Central, last updated December 10, 2025. Stripped of e-commerce framing, the core rules apply to any paginated collection:

• Link every page in the series with real anchor elements, not JavaScript click handlers. Google's crawlers don't click buttons and generally don't trigger JavaScript that requires a user action.
• Give each page a unique, stable URL, and use that same URL consistently in internal links, your sitemap, and canonical tags — Google specifically calls out inconsistent pagination URLs as the most common mistake it sees.
• Canonicalize each page to itself, not to page 1. Collapsing page 2 through page 40 onto the canonical of page 1 tells Google (and, by extension, anything that trusts the canonical tag) that only page 1 exists.
• Don't encode the page number in a URL fragment. Google ignores fragment identifiers when discovering pages, so a fragment-based pager is functionally invisible to it.
• Duplicate titles and meta descriptions across a paginated series are fine — Google explicitly does not penalize this, so don't waste effort artificially varying them.

For infinite scroll specifically, Google's separate guidance — first laid out in a 2014 Search Central post on infinite-scroll-friendly design and reiterated in its current lazy-loading documentation — is to chunk the feed into component pages, each with its own persistent URL, and update the browser URL via the History API as the user scrolls. The scroll itself can stay; what has to exist underneath it is a normal paginated structure a crawler can walk without ever firing a scroll event.

Why AI crawlers make this less forgiving, not more

Direct answer: If your collection already survives Googlebot, it's tempting to assume it survives AI retrieval too. It often doesn't, because the constraint that broke infinite scroll for Google — no scrolling, no clicking — is stricter for most AI crawlers, not looser.

Vercel's analysis of AI crawler traffic, published December 17, 2024, found that none of the major AI crawlers — OpenAI's, Anthropic's, Meta's, ByteDance's, or Perplexity's — render JavaScript. ChatGPT's crawler fetched HTML on 57.70% of requests and JavaScript files on 11.50% (fetched, not executed); Claude's crawler spent 35.17% of fetches on images and 23.84% on JavaScript it likewise doesn't run. In the same dataset, ChatGPT's crawler hit 404 pages on 34.82% of fetches and Claude's on 34.16% — roughly four times Googlebot's 8.22% 404 rate in the same comparison — a sign these crawlers are working from stale or malformed link sets more often than Google is. GPTBot alone generated 569 million requests in the period Vercel measured, Claude's crawlers 370 million, and PerplexityBot 24.4 million: this is not marginal traffic, and none of it can see content that only exists behind a "Load more" click or a scroll-triggered fetch.

The practical upshot: if item 47 in your directory only becomes reachable by clicking "Load more" nine times, it is invisible to Googlebot, invisible to GPTBot and ClaudeBot, and invisible to any real-time answer engine trying to synthesize a response — regardless of how good the content is once you get there.

Comparison: pagination patterns and what actually gets crawled

The pattern that survives both regimes is the hybrid: numbered, anchor-linked pages (or a view-all page for smaller sets) with any scroll or "load more" behavior layered on top as progressive enhancement, never as the only path to the content.

Step-by-step: structuring a collection so buried items stay retrievable

1. Give every page a persistent, unique URL. A query parameter or a path segment both work; a URL fragment does not, because Google ignores fragments when discovering new pages.
2. Render pagination links as real anchor elements in the initial HTML, not as onClick handlers that fetch content via JavaScript. If you use a JS framework, confirm the href is present in the server-rendered or statically generated markup, not injected after hydration.
3. Self-canonicalize each page. Page 2's canonical tag should point to page 2, not page 1. Consolidating canonicals to page 1 is the single most common way sites accidentally tell search and AI crawlers that only the first page exists.
4. Link every page back to page 1, and ideally to a few neighboring pages, so the series reads as one connected structure rather than a chain that breaks if any single page fails to crawl.
5. If you use scroll or "load more" for the human experience, pair it with the History API so the visible URL updates to match the chunk being viewed, and make sure that URL is independently fetchable — a crawler landing on it directly, with no prior scrolling, should see the same content a user does.
6. List every paginated URL in your XML sitemap (or an equivalent feed) rather than relying on crawlers to walk from page 1 to page 40 sequentially. This matters even more for AI crawlers, which Vercel's data shows already waste over a third of their fetches on dead or redirected URLs — don't make them guess their way to page 40.
7. Keep the collection below the point where crawl budget becomes the bottleneck. Google's crawl budget guidance for large sites frames budget as a function of a site's overall crawl capacity and demand; a collection with thousands of thin pagination pages competes for that same budget against the rest of your site.
8. For collections small enough to reasonably load in one response (roughly a few hundred items), offer a view-all page as an alternative to deep pagination — it removes the depth problem entirely for both classic and AI crawlers, at the cost of page weight.
9. Verify what a crawler actually sees, not what a browser shows you. Fetch a deep page (say, page 30) with JavaScript disabled, or inspect your server logs for the user agents of GPTBot, ClaudeBot, and Googlebot to confirm they're reaching pages past the first one — don't assume correct markup means successful crawling.

What this doesn't guarantee

Direct answer: Fixing pagination makes buried items reachable. It does not make them retrieved or cited. A few limits worth naming explicitly:

• Reachability isn't ranking. Google's own documentation notes that pages deep in a paginated series compete on their own merits once crawled — a correctly structured page 40 is still a page 40, competing against page 1 for relevance to any given query.
• Crawl budget is finite. Even with clean URLs and sitemaps, Google's crawl budget guidance is explicit that very large sites won't get every URL crawled on every pass; extremely deep collections (hundreds of pages) may still lag in freshness even when fully crawlable.
• AI answer engines choose what to synthesize, and that choice isn't fully observable. A real-time retrieval crawler reaching your page 12 doesn't guarantee that page gets quoted in a generated answer; retrieval and citation are separate steps, and the second one isn't governed by the technical guidance in this article.
• This doesn't fix content quality problems. Thin, near-duplicate pagination pages with little unique text per page remain a weak result even when perfectly crawlable — pagination hygiene and content depth are separate problems that both need solving.

Where nqzai fits

nqzai runs SEO, AEO, and GEO research in the background rather than requiring a separate technical audit tool, and part of that background work is checking whether a site's own structure is quietly hiding content — including whether paginated collections rely on JavaScript-only navigation, whether deep pages are missing from the sitemap, and whether canonical tags are accidentally collapsing a whole series onto page 1. Because it's conversational, you can ask it directly which of your collection pages are reachable only through a "load more" click, and it surfaces the answer instead of you having to run a separate crawl and cross-reference it by hand.

FAQ

Does Google still support rel="next" and rel="prev"?

No. Google confirmed in March 2019 that it had stopped using rel=next/prev as an indexing signal, and its original 2011 blog post introducing the markup now carries an editor's note pointing to current guidance instead. Leaving the tags in your code doesn't hurt — other engines like Bing still read them — but adding them today has no effect on Google or AI crawlers.

Is infinite scroll always bad for SEO and AI visibility?

Pure, JavaScript-only infinite scroll is bad for both, because neither Googlebot nor any major AI crawler scrolls or clicks to load more content. A hybrid implementation — scroll for users, real per-chunk URLs updated via the History API underneath — avoids the problem entirely and is the pattern Google has recommended since 2014.

Should I switch to a "view all" page instead of numbered pagination?

For collections of a few hundred items or fewer, a view-all page removes the depth problem completely, since one fetch retrieves everything. For collections in the thousands, a single page becomes too heavy to load reliably, and numbered, anchor-linked pagination is the more reliable structure.

How does an AI answer engine find something buried on page 12 of a listing?

The same way a search engine does: by following a crawlable link from page 11 to page 12, or by finding page 12's URL directly in a sitemap. If page 12 only exists after nine clicks of "load more," most AI crawlers — which, per Vercel's traffic analysis, don't render JavaScript any more than Googlebot does — simply never reach it.

Do I still need an XML sitemap if my pagination links are already crawlable?

Yes, especially for AI crawlers. Sitemaps let a crawler jump straight to page 40 instead of walking there sequentially, which matters given how much of AI crawler traffic is already spent on dead or redirected URLs rather than useful content.