Skip to content

feat(copy): use rsync by default with scp fallback#423

Open
theFong wants to merge 9 commits into
mainfrom
feat/copy-rsync-fallback
Open

feat(copy): use rsync by default with scp fallback#423
theFong wants to merge 9 commits into
mainfrom
feat/copy-rsync-fallback

Conversation

@theFong

@theFong theFong commented Jul 14, 2026

Copy link
Copy Markdown
Member

Continuation of #297 by @immanuel-peter (his commits are preserved on this branch), rebased onto current main since the fork branch had conflicts.

Summary

  • brev copy now attempts rsync first for file/directory transfers and falls back to scp automatically
  • External node copies (copyExternalNode) go through the same rsync-first path
  • If rsync is missing locally, we skip straight to scp and print a hint to install rsync for faster transfers
  • If rsync is missing on the instance, the fallback message says so and suggests installing it there
  • Tests cover arg construction and all fallback paths

Why rsync first

rsync supports incremental/delta transfer, so repeated and large transfers send fewer bytes and finish faster than scp. scp remains as an automatic fallback so environments without working rsync behave as before.

Verification

  • go test ./pkg/cmd/copy
  • go build ./...
  • golangci-lint v2: 0 issues

Closes #297

Performance: rsync vs scp

Measured (real Brev instance)

Benchmarked from macOS (openrsync, protocol 29) to a running Brev instance (n2d-standard-2, Ubuntu 22.04, GNU rsync 3.2.7) over a real WAN link (~1 MB/s upload at test time), using the exact commands this PR ships: scp [-r] vs rsync -z -e ssh [-r]. All transfers md5-verified after the run.

Scenario scp rsync -z Speedup
20 MB incompressible (random bytes), first copy 29.6s 29.8s ~1× (wash)
20 MB compressible text (logs), first copy 17.5s 0.6s ~28×
Source tree: 248 files, 1.8 MB, first copy 125.3s 0.7s ~187×
20 MB file re-sent after 1% (200 KB) changed 7.4s 1.7s ~4×
20 MB file re-sent, unchanged 8.5s 1.0s ~9×

Notes on the numbers:

  • The many-small-files case is where scp hurts most: scp pays per-file round trips (~125s to move 1.8 MB!), while rsync pipelines the whole tree over one stream.
  • Re-transfer wins come from rsync's delta algorithm and quick-check (size+mtime) skip — scp always recopies everything.
  • The link sped up mid-run (scp rows 4–5 moved 20 MB in ~8s vs ~30s in row 1), so per-row speedups are conservative where the faster link favored scp.
  • First-time copy of a single incompressible file is a wash — as expected, and consistent with public benchmarks.
  • This also validates macOS openrsync ↔ GNU rsync 3.2.7 interop (incl. -z and delta), which matters since macOS Sequoia 15.4+ ships openrsync instead of GNU rsync (details).

Public references

  • rsync's delta-transfer algorithm sends only differences between source and destination; unchanged files are skipped entirely via the quick-check algorithm (rsync man page). The original Tridgell & Mackerras tech report measured ~5% of file bytes transferred when syncing Linux kernel source tarballs (~24 MB → ~1.1 MB sent).
  • Independent benchmark (1 Gbps, server-to-server): 10 GB as 100k small files — rsync ~30–37% faster than scp; 50 GB single file — roughly comparable (wehaveservers.com).
  • scp cannot resume an interrupted transfer; rsync retransfers only missing data.
  • OpenSSH deprecated the legacy scp protocol in 9.0 (scp now speaks SFTP by default) due to security issues with remote wildcard expansion (OpenSSH 9.0 release notes) — scp is the legacy path going forward.

Known tradeoff

-z compression is a win on slow links with compressible data (code, logs, CSVs) but is CPU-bound and can bottleneck fast links when the payload is already incompressible (model checkpoints, archives, images) — rsync itself keeps a --skip-compress suffix list for this reason (man page, example report). In our test it was neutral even on incompressible data over a ~1 MB/s WAN, which matches the typical laptop→cloud path for brev copy. If users report slow checkpoint uploads on fast links, dropping -z or adding --skip-compress defaults is a easy follow-up.

End-to-end validation

Validated with a binary built from this branch (brev copy, not raw rsync/scp) against a live Brev instance (macOS openrsync client → Ubuntu 22.04 / GNU rsync 3.2.7):

  • ✅ File upload, directory upload, and download — all md5-verified
  • ✅ Unchanged re-send completes in <0.5s (rsync quick-check)
  • ✅ With rsync removed from PATH: prints rsync not found on this machine, using scp. Install rsync for faster transfers. and completes via scp (md5-verified)
  • ✅ Both-tools-fail path produces a combined error showing both rsync and scp output
  • ✅ Instance auto-start + SSH wait flow unaffected

Validation caught one regression, fixed in d4fc346: without a trailing slash, rsync nests an uploaded directory inside the destination (dest/dir/...) where scp made dest itself the copy. Directory-upload sources are now normalized to trailing-slash form so the destination always mirrors the source.

Unified directory semantics (71d7760)

rsync and scp historically disagree about directory copies (nest vs. copy, depending on trailing slashes and whether the destination exists). This PR unifies both engines on one rule: the destination always mirrors the source.

brev copy ./dir instance:destdest/a.txt — same result whether dest exists or not, whether rsync or the scp fallback ran, and for downloads too. Repeated copies are idempotent (no more dest/dir/dir nesting on re-runs).

Implementation: directory sources are normalized to contents-copy form — trailing / for rsync, /. for scp (verified equivalent on OpenSSH 10 SFTP-mode scp). Uploads stat the local source; downloads determine the remote path type with a single ssh <alias> test -d <path> probe (~1 extra round trip per download). All six layout combinations validated end-to-end against a live instance.

immanuel-peter and others added 5 commits February 26, 2026 10:40
# Conflicts:
#	pkg/cmd/copy/copy.go
#	pkg/cmd/copy/copy_test.go
Skip the rsync attempt entirely when rsync is not installed locally,
and when the instance itself lacks rsync, say so in the fallback
message. Both scp paths now tell the user to install rsync for
faster transfers.
@theFong theFong requested a review from a team as a code owner July 14, 2026 16:04
stephahart
stephahart previously approved these changes Jul 14, 2026
Without a trailing slash rsync nests the source directory inside the
destination (dest/dir/...), while scp makes the destination the copy
when it does not exist. Normalize directory-upload sources to
trailing-slash form so the destination always mirrors the source.
theFong added 2 commits July 14, 2026 22:23
Directory sources are normalized to contents-copy form (trailing slash
for rsync, /. for scp) so the destination always becomes a mirror of
the source, whether or not it already exists, and both transfer tools
produce identical layouts. Downloads learn the remote path type via a
single 'ssh test -d' probe; uploads stat the local source directly.
This also makes repeated directory copies idempotent instead of
nesting dir inside dir on the second run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants