Skip to content

fix(e2e): capped/batched receive tests no longer time out on tenant history#57

Merged
andinux merged 1 commit into
mainfrom
fix/chunked-capped-receive-predrain
Jul 16, 2026
Merged

fix(e2e): capped/batched receive tests no longer time out on tenant history#57
andinux merged 1 commit into
mainfrom
fix/chunked-capped-receive-predrain

Conversation

@andinux

@andinux andinux commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • The Chunked Capped Receive test had become permanently red. Root cause: its receiver is a fresh site, so its first check makes the server spool the entire tenant history, which grows ~6 MB with every run (three incompressible-batch tests). The 80×1-chunk budget was spent on ~25 attempts of 202s (server building the spool) + ~55 attempts consuming history pages — and the spool snapshot predates the batch the test just sent, so those pages contained zero rows of it. The test was O(total tenant history) with a fixed budget and inevitably flipped to failing once history outgrew it.
  • A naive pre-drain doesn't work: on a 202 the client reports rows=0, complete=1, indistinguishable from "already drained" (verified via network trace — the pre-drain exited after a single 202).
  • Fix: chunked_receiver_catch_up() — the sender sends a tiny sentinel row and the receiver drains with uncapped receives until the sentinel arrives locally, the only reliable end-of-replay signal. Only then is the test batch sent, making the capped loop O(this test's batch). Applied to both the capped and batched receive tests (the batched one had the same latent failure mode, just a 2× larger effective budget). Sentinels are deleted in the tests' existing remote cleanup.

Testing

  • make e2e: Chunked Capped Receive went from consistently FAILED (3/3 runs, also failing on unmodified main) to OK in both verification runs; Batched Receive OK in both.
  • Diagnosed against full NETWORK_TRACE=1 runs before and after.

Notes

  • The 5-thread Peer Test flakes occasionally with receive.rows=0 for the same underlying reason (2.5 s poll budget vs. main-tenant history spool under 5 concurrent fresh sites); left out of scope here.
  • Tenant history growth is unbounded — a periodic server-side cleanup of the chunked/main test tenants would keep e2e wall-time from creeping up, independent of this fix.

🤖 Generated with Claude Code

…istory

A fresh receiver's first check makes the server spool the ENTIRE tenant
history (which grows every run) and answer 202 while preparing; the spool
snapshot also predates the batch this test just sent. The capped loop's
80x1-chunk budget was therefore O(total history) and started failing
permanently once history outgrew it.

Catch the receiver up before sending the test batch: send a sentinel row
and drain (uncapped) until it arrives. A 202 yields rows=0/complete=1 —
indistinguishable from drained — so the sentinel's arrival is the only
reliable end-of-replay signal. The capped budgets are now O(this test's
batch). Sentinels are deleted in the tests' remote cleanup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@andinux
andinux merged commit 6d34e85 into main Jul 16, 2026
32 checks passed
@andinux
andinux deleted the fix/chunked-capped-receive-predrain branch July 16, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant