Skip to content

Remote execution: SSH submission, job tracking, watch loop, sharded labeling#2

Open
basillicus wants to merge 15 commits into
mainfrom
remote-execution
Open

Remote execution: SSH submission, job tracking, watch loop, sharded labeling#2
basillicus wants to merge 15 commits into
mainfrom
remote-execution

Conversation

@basillicus

@basillicus basillicus commented Jul 12, 2026

Copy link
Copy Markdown
Owner

What

Single-cluster remote execution on top of the existing Slurm executor:

  1. SSH remote submission — cluster profiles (~/.traincraft/clusters/*.toml) gain host (SSH alias) + remote_root: traincraft submit now works from a laptop/VM. The run lives at remote_root/<run.name> on the cluster (a resolved config snapshot + referenced input files are shipped there); the local run dir becomes a mirror (rendered scripts, submission events, pulled state). $SCRATCH/~ in remote_root expand on the remote shell, never locally.
  2. Job-state tracking — per-stage submitted events with job ids; traincraft status gains a queued state. Submission uses --kill-on-invalid-dep=yes so a failed stage takes its dependents down instead of leaving them pending forever.
  3. traincraft watch <config|run-dir> [--interval N] [--once] — re-entrant poll loop (no daemon): ingests the remote events.jsonl + labeling receipts into the mirror (exact-line dedup), polls sacct (job-array aware), and records failed events for jobs the queue killed (OOM/timeout/cancel) that never got to log their own failure. Exit 0/1/3.
  4. Sharded labeling — [orchestration.slurm.stages.label] shards = N renders a Slurm job array (traincraft stage label CONFIG --shard I:N, round-robin frames, per-shard single-writer journal dirs under labeled_dft/shard_XX/) plus a small merge job (--merge-shards: assembles journals by fingerprint, records misses as failures, never recomputes DFT). Retry = resubmit shards; the journals make reruns incremental. shards is execution-only config and never invalidates the label cache key.

Design invariants kept

  • Single writer per file: remote events.jsonl is written only by remote stage processes; the mirror only by local submit/watch (which appends a failure only once the job is provably dead). Shard tasks write no events — receipts are their progress record.
  • Never O(n_frames) files: shards are bounded by worker count.
  • Backward compatible: no host → identical local sbatch flow; shards unset → renderer output unchanged; engine="local" untouched. Existing test_slurm_executor.py assertions pass unmodified.
  • Multi-cluster seams left (follow-up PR): Transport is constructed per cluster config; the watch pass is ingest → poll → act; the snapshot shipper is per-(cluster, run).

Testing

  • 270 passed / 27 skipped (dev env), ruff clean, mkdocs clean.
  • New fake ssh/scp/sbatch/sacct shim harness (tests/conftest.py) runs the real code paths end-to-end against a tmp-path "cluster" — no scheduler or network.
  • Live CLI smoke test: submitted a sharded EMT pipeline through the fake cluster, executed the shipped snapshot's stages exactly as the sbatch scripts would (--shard 0:2/1:2/--merge-shards), watched it to run completed. The smoke test caught two real bugs (stale stage error text surviving a rerun; the merge job leaking into planned stages) — both fixed with regression tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS

David Abbasi Perez and others added 15 commits July 7, 2026 15:39
@claude
feat(labeling): per-frame failure isolation (on_error skip/raise), qu…
e648f48 
…arantined failures, and receipt-based resume that reuses completed frames and retries failed ones

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
feat(validation): parity metrics + thresholded quality_report.json as…
cf1c881 
… a 'validate' stage (defaults to the trained model and its holdout valid.xyz); decode array properties in read_frames

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
docs: teach the skill, CLI/API reference, examples (26) and tutorials…
330dc45 
… the new validate stage and labeling failure semantics

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
feat(labeling): replace per-frame directories with append-only receip…
17151c9 
…ts/journal/quarantine files so 100k+-frame runs stay Lustre-safe

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
feat(state): run-state layer — per-run events.jsonl + lazy SQLite ind…
1991265 
…ex, traincraft runs/status commands

- stages emit start/done/skipped/failed events (one small events.jsonl per run,
  appended by whichever process runs the stage; safe on Lustre because stage
  jobs are dependency-chained and appends need no locking)
- RunIndex: disposable SQLite cache (<outdir>/index.db) refreshed lazily by the
  CLI on the login node only; delete-safe, rebuilt from event logs
- traincraft runs [outdir] lists all runs; traincraft status <run-dir> shows
  per-stage state with live labeling progress read from receipts.jsonl
- run_status/RunIndex exported in the public API (the base for the MCP/FastAPI
  service and the workbench)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
feat(service): HTTP (FastAPI) + MCP service over the run-state layer
be9e8cb 
- service/core.py: shared, dependency-free ops (list/status/events/report/
  manifest, TOML validation, submission — Slurm or detached local process)
- traincraft serve: localhost FastAPI app with OpenAPI docs at /docs
- traincraft mcp: the same six operations as MCP tools on stdio, with working
  rules (validate before submit, judge models only by the quality report)
  encoded in the tool docstrings
- optional deps behind 'pixi install -e service'; new how-to page + CLI/API
  reference; roadmap updated

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
polish(cli): rich help with workflow-grouped command panels and per-c…
9a32536 
…ommand examples, colored runs/status output, --json for the inspect commands

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
feat(geometry): multi-structure spine — build_geometries, [geometry] …
9e9daad 
…replicate, multi-frame file/conformer sources, every seed sampled

The geometry stage now emits N seed structures and the sampling stage samples
all of them (per-seed job dirs, pooled candidates, parent hash in provenance)
instead of silently dropping everything but frame 0. Producers: replicate=N
with derived per-replica seeds, smiles n_conformers>1, file index=":".
build_geometry stays as the strict exactly-one convenience. Prerequisite for
diffusion-model sources.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6f4Z4uLdysBeehYfbQ56k
@claude
feat(state): submitted/queued job events — accumulate-on-start fold, …
a48e281 
…queued in the status ladder, shard-aware label progress

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@claude
feat(orchestration): Transport seam — run/shell/push/pull locally or …
0f208f6 
…over an SSH alias, with a fake ssh/scp/sbatch/sacct test harness

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@claude
feat(slurm): SSH remote submission — host/remote_root on profiles, re…
252c33e 
…solved config snapshot + inputs shipped to the cluster, per-stage submitted events, kill-on-invalid-dep

The local run dir becomes a mirror (scripts, submission events); the remote
run dir at remote_root/<run.name> is the truth the jobs write. dumps_config
round-trips the snapshot (exclude_none only: discriminator type fields are
Literal defaults and must survive).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@claude
feat(watch): traincraft watch — sacct polling with array aggregation,…
06296b4 
… remote events/receipts ingest into the local mirror, failed events for queue-killed jobs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@claude
feat(label): sharded labeling — stage --shard I:N / --merge-shards, S…
8054d42 
…lurm job array + merge job, per-shard single-writer journal dirs

Shard tasks label frames[i::n] into labeled_dft/shard_XX/ (own work/,
receipts, journal — the single-writer invariant holds per dir) and publish
nothing; the merge job assembles journals by fingerprint, records misses as
failures (never recomputes DFT in its small allocation), and emits the
stage's events/artifact/key. shards lives on SlurmStage so changing it never
invalidates the label cache key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@claude
fix(state,slurm): a new attempt clears a stage's stale error text, an…
5ae5dfa 
…d the label_merge job stays out of the planned-stages list (found by live fake-cluster smoke test)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@claude
docs: SSH-mode submission, traincraft watch, sharded labeling — hpc h…
1fe28bb 
…ow-to, config/cli reference, agent skill recipes, roadmap tick + multi-cluster follow-up

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RmprSjfXNtRV84X9cdkwS
@basillicus basillicus assigned basillicus and unassigned basillicus Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@basillicus basillicus

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@basillicus