Skip to content

checkpoint: cut live restore over to the execve stub and remove the injection engine #145

Description

@congwang-mk

Summary

Wire the freestanding execve restore-stub (already landed, tested, but #[allow(dead_code)]) into the live restore path, and replace the ptrace injection engine with it. The stub restores a checkpoint into a fresh execve'd address space that holds only the checkpoint image plus a fresh kernel vDSO/vvar, eliminating the leftover-launcher-mapping contamination that limits the current injection engine.

Background

Live-process restore today (Sandbox::restore_interactive -> checkpoint::resume::restore_into) works by forking a child that parks before execve, then ptrace-injecting the checkpoint image over that parked launcher. This works for an idle libc program (file-backed maps, anon data via process_vm_writev, fd reopen, and [vdso]/[vvar] relocation via plan_vdso_moves are all handled), but a busy workload segfaults on resume: the parked launcher's own mappings (libc, stack, unswept trampolines) coexist with the injected image and get hit once the restored program runs freely.

The execve-stub apparatus was landed incrementally to fix this at the root by giving restore a clean address space:

  • checkpoint/restore_blob.rs - control-blob wire format + serializer (anon-only today; n_fds hardcoded 0; file-backed regions marked NO_DATA).
  • checkpoint/restore-stub.c - freestanding x86_64 stub: maps anon regions MAP_FIXED, registers userfaultfd (missing mode), READY/GO handshake with the supervisor, rt_sigreturns into the checkpoint registers. No file-backed regions, no fd table, no vDSO relocation, no FP restore yet.
  • checkpoint/pager.rs - supervisor-side userfaultfd page server for the anon working set.
  • tests/integration/test_restore_stub.rs - end-to-end proof with a hand-assembled single-region program.

All of the above is exercised only by its own unit/integration tests and is not called from restore_interactive/restore_into. See the checkpoint/mod.rs comment ("not yet wired into the live restore code ... drop [the allow] then").

Goal (first milestone)

Restore a real, idle libc program via the stub (concrete target: the musl sleep-loop service process we already restore via injection). Definition of done: checkpoint the idle service, restore it through the stub path, and observe it resume and keep running with correct clock_gettime/vDSO behavior. The busy-workload case is explicitly out of scope for this milestone (it is the next milestone the clean address space unlocks).

Approach: pure execve stub

The stub reconstructs the whole address space itself; the supervisor only serves anon pages lazily over userfaultfd.

Stub (restore-stub.c) gains:

  1. File-backed regions - openat the region's in-chroot path and mmap MAP_PRIVATE|MAP_FIXED at file_off with the recorded prot. Writable-private file-backed pages that were dirtied are captured as anon data and overlaid (uffd-served) on top.
  2. fd table - reopen each restorable fd: openat the in-chroot path with saved flags, lseek to the saved offset, dup2 to the target fd number.
  3. vDSO/vvar relocation - find the fresh vDSO via AT_SYSINFO_EHDR in auxv and mremap [vdso] (and [vvar]/[vvar_vclock]) onto the checkpoint-recorded base, so glibc/musl's cached vDSO pointers resolve. (vvar has no auxv entry; locate it relative to vDSO / from a supervisor-passed base if needed.)
  4. FP register restore - populate the rt_sigframe fpstate from the captured fpregs instead of the current m->fpstate = 0.

Blob format (restore_blob.rs) gains (bump BLOB_VERSION):

  • Real fd table entries (n_fds > 0): fd number, in-chroot path, flags, offset.
  • Per file-backed region: the in-chroot path string + file offset.
  • memfd / (deleted)-backed private regions (e.g. /memfd:sandlock-exec (deleted)) marked anon-with-data so their captured bytes are paged in (this pairs with the capture-side dump already added in the chroot-restore fix).

Supervisor (restore_interactive / a new resume entrypoint): fork child, apply confinement (chroot + Landlock + seccomp), install the fixed inherited fds (control-blob memfd at CTRL_FD, READY_FD/GO_FD eventfds, UFFD_SLOT), execve the stub, pidfd_getfd the uffd, run the pager, GO, wait.

Remove the duplicated logic

Once the stub reaches parity on the idle case, delete the injection engine and de-duplicate:

  • Remove checkpoint/inject.rs and the ptrace-injection path in checkpoint/resume.rs (restore_into, RemapFromFile/WriteBytes application, injected fd reopen).
  • Drop the #[allow(dead_code)] markers on restore_blob/pager/the CTRL_FD/READY_FD/GO_FD/UFFD_SLOT constants in checkpoint/mod.rs.
  • Single home for shared planning. The region/fd/vDSO planning logic (build_memory_plan, build_fd_plan, plan_vdso_moves, and the host->in-chroot path translation chroot::resolve::host_to_virtual) should live in exactly one place and feed the blob serializer, rather than being re-implemented in the stub. Keep the decisions (what to map, which fds are restorable, which paths translate) in Rust at serialize time; the stub stays a dumb executor of the blob. Anything in resume.rs that survives only to serve injection goes away with it.

Confinement / seccomp consideration

The stub runs inside the sandbox (confinement must be applied before execve because rt_sigreturn hands control to the restored program with no later setup hook). The seccomp policy must therefore permit the stub's syscalls during restore: openat, mmap, mremap, lseek, fstat, ioctl, userfaultfd, dup2, read, write, rt_sigreturn, exit. Decide whether these are always-allowed by policy or gated behind a restore-time allowance (note: seccomp is one-way, so anything the stub needs, the restored program keeps). Landlock must grant read/exec on the image files at their in-chroot paths; chroot must already be in place so those paths resolve.

Out of scope (future milestones)

  • Busy-workload restore (the clean address space is the prerequisite; fidelity of an actively-running register/stack/vdso state is its own milestone).
  • Non-x86_64 architectures (stub is x86_64-only; keep the existing arch guard).
  • Sockets/pipes/eventfd fds (remain surfaced via restore_skipped / the app_state hatch).

Acceptance

  • Stub restores the idle musl service end-to-end; process resumes and survives (running == true), vDSO calls work.
  • Injection engine (inject.rs + inject path in resume.rs) removed; no #[allow(dead_code)] left on the stub machinery.
  • Shared planning logic lives in one place; stub carries no duplicated decision logic.
  • Existing resume/chroot-restore tests still pass; a new end-to-end stub-restore test covers the idle libc case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions