pipeline: transactional sequential pipelines over a shared COW workdir (RFC #65 Phase 1)#148
Open
dzerik wants to merge 3 commits into
Open
pipeline: transactional sequential pipelines over a shared COW workdir (RFC #65 Phase 1)#148dzerik wants to merge 3 commits into
dzerik wants to merge 3 commits into
Conversation
…ikernel#65 Phase 1) Add Pipeline::run_transactional(): sequential stages share one COW upper over a common workdir, so a later stage sees an earlier stage's writes (read-committed), the real workdir is untouched during the run, and the pipeline commits all-or-nothing -- every stage exits 0 -> one commit; any stage fails -> the shared upper is discarded and the workdir is byte-identical. Stages exchange data through the shared workspace, not inter-stage pipes; the streaming run()/`|` path is unchanged. Sequential stages let one shared upper stand in for the RFC's per-stage lower-lookup chain (the shared SeccompCowBranch is injected into each stage's runtime; wait()/Drop leave it for the coordinator's single commit/abort). Per-stage uppers (parallel siblings) are Phase 2. Hardening from an adversarial self-review: - Add `impl Drop for SeccompCowBranch` (cleanup unless finished) so an aborted- by-error/timeout/panic transaction -- or any abandoned branch -- can't orphan its upper on disk. `keep()` marks BranchAction::Keep so the backstop preserves a kept upper instead of cleaning it. - run_pipeline_txn finalizes (commit on clean success, else abort) on EVERY exit path, including a driver error, before propagating it -- the branch is never left dangling; the branch is taken out from under the async mutex before the sync merge. - Guardrails: reject a <2-stage pipeline (no stages[0] panic), a stage with chroot, or a stage whose fs_storage/max_disk differs from stage 0 (all share one upper). canonicalize the workdir before creating the branch dir so a failure there can't leak an empty dir. TxnOutcome derives Debug/Clone. Integration tests cover commit-on-success, abort-on-failure (no leak), timeout-abort, and the guardrails (pinned to the error message); a unit test covers the Drop backstop and keep() preservation.
The initial suite tested 3 of 6 guardrails and only the last-stage-failure / clean-abort/commit paths. Add the missing cases so a regression in any guardrail or boundary hard-fails instead of slipping through: Guardrails (no sandbox needed): - mismatched workdir across stages (distinct from missing workdir) - no_supervisor=true stage - chroot stage - mismatched fs_storage/max_disk across stages Behavioral (sandbox-gated): - first-stage failure aborts AND stops early (outcome.stages holds only the failed stage — later stages must not run) - timeout aborts AND reclaims the shared upper (the reclaim test only covered clean abort/commit) - COW deletion (whiteout): commit applies the deletion and a later stage sees it (read-committed); abort discards it and the file survives byte-identical All 14 pass on a real cage.
…abort Clarify that stages only carries per-stage results on a stage-failure abort; on a timeout or driver-error abort the stage-driver future is cancelled and its accumulated results are dropped, so the field is empty while committed and abort_reason still report the outcome.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Phase 1 of RFC #65: transactional sequential pipelines.
What
Pipeline::run_transactional()— sequential stages share a single COW upper over theworkdir. The real workdir is untouched during the run; the pipeline commits atomically
(all-or-nothing) on clean success and aborts (discards the upper) on any error/timeout/panic.
Design note (per the thread)
The RFC sketched a per-stage multi-layer lower-lookup chain (
[own upper, stage_N upper, …, workdir]), butSeccompCowBranchis strictly two-layer (oneworkdirlower + one upper).Phase 1 therefore uses one shared upper across the sequential stages instead of stacking a
layer per stage — the behavior you green-lit on 2026-07-13.
Guardrails & tests (14, all green)
run_pipeline_txnfinalizes on every exit path (commit on clean success, else abort) —an error/timeout/panic transaction, or any abandoned branch, can't orphan an upper.
<2-stage pipeline (nostages[0]panic), empty-stage boundaries, etc.cargo test txn→ 14 passed; rebased on currentmain(builds clean).Scope — deferred, documented (not a regression)
one workdir each build a private upper and merge file-by-file at commit with no coordination,
so they interleave (last-writer-wins) rather than the first-committer-win semantics. Per-
transaction all-or-nothing still holds; cross-transaction serialization is out of scope for
Phase 1 and left as a documented limitation (this is the open scope question from the thread).
general COW issue, not introduced here — filed and fixed separately in fix(cow): return ENOENT when opening a file deleted in the branch #144.
Refs #65.