Skip to content

perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops#2220

Draft
purp wants to merge 1 commit into
NVIDIA:mainfrom
purp:2219-tcp-nodelay/purp
Draft

perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops#2220
purp wants to merge 1 commit into
NVIDIA:mainfrom
purp:2219-tcp-nodelay/purp

Conversation

@purp

@purp purp commented Jul 10, 2026

Copy link
Copy Markdown

Summary

Every small request/response through the sandbox relay path paid a fixed
~44 ms penalty because no socket in the path disabled Nagle's algorithm, so
sub-MSS ("tinygram") writes stalled waiting on TCP delayed ACKs at each hop.
This sets TCP_NODELAY on every latency-sensitive TCP socket in the tunnel
and egress-proxy path, cutting the marginal cost of a small governed
request from ~44 ms to ~0.3–0.5 ms — within a few hundred microseconds of
the host loopback floor — with no regression on bulk transfers.

Opening as a draft to invite discussion of the approach before it's marked
ready.

Related Issue

Fixes #2219

Changes

  • gateway: set TCP_NODELAY on accepted connections on the public
    listener (gRPC relay frames and WS tunnel writes).
  • CLI: edge-tunnel local accept + the WebSocket's underlying TCP
    stream, the insecure TLS connector (matching tonic's default connector),
    and service-forward accepted sockets.
  • supervisor: direct-tcpip connect into the sandbox netns, TCP relay
    target dials (connect_tcp_target), egress-proxy accepted connections,
    and all upstream CONNECT/HTTP dials (via a new connect_upstream helper).
  • Setting TCP_NODELAY on connect is best-effort: a failure only costs
    latency, so we log and continue rather than fail the connection.
  • Regression tests assert TCP_NODELAY is set on the proxy upstream,
    direct-tcpip, and TCP-relay connect paths.

The sandbox SSH transport rides a unix domain socket and gRPC client
channels use tonic defaults (nodelay on), so no change is needed there.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated (nodelay regression tests for the three
    connect hops)
  • E2E tests added/updated (if applicable) — not applicable; validated
    with a component-isolation benchmark ladder instead (below)

A/B benchmark ladder (medians over 30 reps/cell; macOS arm64, Docker
compute driver). Tools and full results are on a temporary branch:
relay-latency-benchmarking.

Keep-alive marginal cost per small request:

Path baseline this PR host floor
L4 passthrough, 100-req conn 43.58 ms 0.26 ms 0.06 ms
L7 terminate+inspect, 100-req conn 44.49 ms 0.45 ms 0.06 ms

No regression (worst cases): 10 MB response 58.6 → 59.0 ms (unchanged);
10 MB request 90.4 → 46.0 ms (Nagle-throttled uploads got ~2× faster);
L7 body-rewrite rung 1 KB 63.7 → 14.9 ms.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable) — not applicable; this is a
    socket-option tuning with no change to subsystem boundaries or data flow

The sandbox tunnel added ~44 ms of latency to every small
request/response because no socket in the path disabled Nagle's
algorithm, so sub-MSS writes waited on delayed ACKs at each hop.

Set TCP_NODELAY on every latency-sensitive TCP socket:

- gateway: accepted connections on the public listener (gRPC relay
  frames and WS tunnel writes)
- CLI: edge tunnel local accept + underlying WebSocket TCP stream,
  insecure TLS connector (tonic's default connector already does
  this), and service-forward accepted sockets
- supervisor: direct-tcpip connect into the sandbox netns, TCP relay
  target dials, egress proxy accepted connections, and all upstream
  CONNECT/HTTP dials (via a new connect_upstream helper)

Setting TCP_NODELAY on connect is best-effort: a failure only costs
latency, so we log and continue rather than fail the connection.

The sandbox SSH transport rides a unix domain socket and gRPC client
channels use tonic defaults (nodelay on), so no change is needed there.

Fixes NVIDIA#2219

Signed-off-by: Jim Meyer <jim@meyer4hire.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

~45 ms added latency on sandbox tunnels/egress tinygrams due to Nagle × delayed-ACK issue

1 participant