fix(e2e): capped/batched receive tests no longer time out on tenant history#57
Merged
Merged
Conversation
…istory A fresh receiver's first check makes the server spool the ENTIRE tenant history (which grows every run) and answer 202 while preparing; the spool snapshot also predates the batch this test just sent. The capped loop's 80x1-chunk budget was therefore O(total history) and started failing permanently once history outgrew it. Catch the receiver up before sending the test batch: send a sentinel row and drain (uncapped) until it arrives. A 202 yields rows=0/complete=1 — indistinguishable from drained — so the sentinel's arrival is the only reliable end-of-replay signal. The capped budgets are now O(this test's batch). Sentinels are deleted in the tests' remote cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
checkmakes the server spool the entire tenant history, which grows ~6 MB with every run (three incompressible-batch tests). The 80×1-chunk budget was spent on ~25 attempts of 202s (server building the spool) + ~55 attempts consuming history pages — and the spool snapshot predates the batch the test just sent, so those pages contained zero rows of it. The test was O(total tenant history) with a fixed budget and inevitably flipped to failing once history outgrew it.rows=0, complete=1, indistinguishable from "already drained" (verified via network trace — the pre-drain exited after a single 202).chunked_receiver_catch_up()— the sender sends a tiny sentinel row and the receiver drains with uncapped receives until the sentinel arrives locally, the only reliable end-of-replay signal. Only then is the test batch sent, making the capped loop O(this test's batch). Applied to both the capped and batched receive tests (the batched one had the same latent failure mode, just a 2× larger effective budget). Sentinels are deleted in the tests' existing remote cleanup.Testing
make e2e: Chunked Capped Receive went from consistently FAILED (3/3 runs, also failing on unmodifiedmain) to OK in both verification runs; Batched Receive OK in both.NETWORK_TRACE=1runs before and after.Notes
receive.rows=0for the same underlying reason (2.5 s poll budget vs. main-tenant history spool under 5 concurrent fresh sites); left out of scope here.🤖 Generated with Claude Code