pagination-normalize

Normalize cursor, offset, and infinite pagination into one duplicate-free cache shape.

Every team that builds an infinite feed re-implements the same merge logic — and most get it subtly wrong. pagination-normalize is the small, framework-agnostic core you actually needed: a normalized paginated collection that stores an ordered list of ids plus an entity map, so overlapping pages update data instead of duplicating rows, and out-of-order pages still render in the right order. Zero runtime dependencies.

Deep-dive guides on these patterns live at frontendcache.com.

---

The problem

Pagination looks trivial until real data hits it. Concatenating pages.flat() — the default in almost every useInfiniteQuery / SWR-infinite tutorial — breaks in ways that only show up in production:

• Cursor overlap. The underlying list mutates between requests, so page 3 comes back overlapping page 2 by two items. pages.flat() renders those items twice, and React logs "Encountered two children with the same key."
• Offset gaps. A user scrolls to the bottom, jumps back up, and now you've loaded rows [0..20) and [40..60) but never [20..40). Flattening silently hides twenty rows with no error at all.
• Out-of-order arrival. A prefetched older page resolves after a newer one. Naive appends put it at the end; the feed renders scrambled.
• Stale re-fetches. The same entity comes back on a later page with a newer score or updated_at. Which copy wins? With a flat array, whichever one you happened to render.

These are not exotic. They are the normal behavior of any list that changes while you paginate it. The fix is always the same shape — normalize to ids + entities and merge by id — so this library just implements that shape correctly once.

---

Install & use

Not published to a registry — clone and build:

git clone https://github.com/frontendcache/pagination-normalize.git
cd pagination-normalize
npm install
npm run build

Then import from the built output:

import { createCollection } from "./dist/index.js";

(Or add the folder as a local file dependency and import pagination-normalize.)

---

Worked example

An infinite feed whose cursor pages overlap — the classic case that produces duplicate React keys:

import { createCollection } from "./dist/index.js";

interface Post {
  id: number;
  title: string;
  score: number;
}

const feed = createCollection<Post>({
  getId: (p) => p.id,
  mode: "cursor",
});

// Page 1
feed.appendPage(
  [
    { id: 1, title: "hello", score: 10 },
    { id: 2, title: "world", score: 8 },
  ],
  { cursor: null, hasMore: true },
);

// Page 2 — the list shifted, so it overlaps page 1 by item 2,
// and item 2 now has a fresher score.
feed.appendPage(
  [
    { id: 2, title: "world", score: 12 },
    { id: 3, title: "again", score: 5 },
  ],
  { cursor: "c2", hasMore: false },
);

feed.ids();          // => [1, 2, 3]   — no duplicate, order preserved
feed.get(2)?.score;  // => 12          — later page updated the entity
feed.size;           // => 3
feed.hasMore;        // => false       — from the last page's meta

No duplicate 2, item order intact, and the freshest data for item 2 — from four lines of appendPage. The merge-by-id logic that removes duplicates is the whole point; you never hand React a repeated key again.

Detecting gaps in an offset-paginated list:

const rows = createCollection({ getId: (r) => r.id, mode: "offset" });

rows.appendPage(firstTwentyRows,  { offset: 0,  limit: 20 });
rows.appendPage(rowsFortyToSixty, { offset: 40, limit: 20 });

rows.gaps();             // => [{ start: 20, end: 40 }]
rows.hasGapBefore(50);   // => true  — you'd cross a hole to reach row 50

---

API reference

createCollection<T>(options)

Option	Type	Default	Notes
getId	(item: T) => string | number	—	Required. Stable, pure id extractor.
mode	'cursor' | 'offset' | 'infinite'	'cursor'	Only changes which helpers make sense; the normalized core is identical.
dedupe	boolean	true	De-duplicate by id on merge.
merge	(existing: T, incoming: T) => T	shallow merge	How to combine two entities with the same id.

Merging pages

• appendPage(items, meta?) — merge a page at the end (next scroll / forward cursor).
• prependPage(items, meta?) — merge a page at the start (older items / backward cursor).
• mergePage(items, { at?, meta? }) — general merge; at is 'start' | 'end' | number. New ids are spliced in at at; ids already present update their stored entity in place and keep their position, so a merge can never reorder or duplicate the list.

meta (PageMeta) is boundary info you supply: cursor, offset, limit, hasMore, pageParam, plus any API-specific fields.

Reading

• items() — denormalized items in render order.
• ids() — the ordered ids (a copy).
• get(id) / has(id) — entity lookup.
• size — number of distinct items.
• hasMore — meta.hasMore of the furthest-forward page.
• pages() — the per-page records (ids + meta), in list order.

Duplicate & gap detection

• findOverlap(items) — ids from items already present in the collection.
• gaps() — missing offset ranges between the lowest and highest loaded offset (offset mode). E.g. loaded [0..20) and [40..60) → [{ start: 20, end: 40 }].
• hasGapBefore(index) — whether a detected gap starts before index.

Staleness & refetch

• markStale(id | id[]) / clearStale(id?) — flag/unflag ids.
• stale(id) — whether an id is flagged (explicitly, or because a later merge changed its value).
• pagesToRefetch() — page records that contain at least one stale id. Pragmatic by design: it reports which loaded pages own changed data; it does not diff the server for you.

Standalone pure helpers

For when you just need to fold a few pages without holding a collection:

• mergePagesById(pages, getId, merge?) — flatten T[][] into one ordered, de-duplicated T[].
• dedupeById(items, getId, merge?) — de-duplicate one list; first occurrence sets position, later data wins on value.
• detectGaps(ranges) — missing ranges between loaded ranges. Also coalesceRanges and isCovered.

React Query InfiniteData adapter

Pure functions, no @tanstack/react-query dependency:

• toInfiniteData(collection, options?) — project the collection into { pages, pageParams }, with overlap already stripped from each page.
• fromInfiniteData(data, options) — build a collection from an InfiniteData object; pageParams round-trip through page meta.

import { fromInfiniteData, toInfiniteData } from "./dist/index.js";

const collection = fromInfiniteData(queryData, { getId: (i) => i.id, mode: "infinite" });
// ...merge realtime updates, dedupe overlap, detect gaps...
const clean = toInfiniteData(collection); // feed back to the query cache

---

Design notes

Why not just concat pages? Because a flat array conflates two concerns — order and identity — that pagination pulls apart. The moment the same id can appear on two pages, or a page can arrive late, order and identity have to be tracked separately. That is exactly an ordered id list plus an entity map. Once you store it that way, dedup is "is this id in the map?", overlap is a set intersection, and a re-fetch is a map write — all O(1)-ish, none of it a linear scan-and-filter over a growing array.

Why keep an entity map at all, not just unique ids? So a later page (or a realtime patch, or an optimistic mutation) can update an item that an earlier page already rendered, without touching the earlier page. The list keeps the item's original position; the map holds its freshest value. Unchanged entities keep their object reference, which keeps React re-renders cheap.

Why is pagesToRefetch() deliberately dumb? Real staleness is a server concern — only your backend knows if page 2 changed. This library tracks the signal it can see locally: an entity whose value changed under an already-loaded page, plus anything you markStale. It hands you the affected page records; you decide what to refetch.

Gap detection is offset-only on purpose. Opaque cursors have no arithmetic — you cannot know how many items sit between two cursors, so there is no honest "gap" to compute. Offsets are numbers, so gaps are well defined. Mixing the two would invent data.

---

Related reading

Deep-dive articles on the normalization and pagination patterns this library implements:

• Data normalization & query key design
• Pagination normalization patterns
• Normalizing cursor-based pagination
• Merging paginated lists without duplicates
• Nested data flattening techniques

---

Development

npm install
npm run build      # tsc -> dist
npm run typecheck  # tsc --noEmit
npm test           # node:test under tsx

License

MIT — Copyright (c) 2026 frontendcache