perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops#2220
Draft
purp wants to merge 1 commit into
Draft
Conversation
The sandbox tunnel added ~44 ms of latency to every small request/response because no socket in the path disabled Nagle's algorithm, so sub-MSS writes waited on delayed ACKs at each hop. Set TCP_NODELAY on every latency-sensitive TCP socket: - gateway: accepted connections on the public listener (gRPC relay frames and WS tunnel writes) - CLI: edge tunnel local accept + underlying WebSocket TCP stream, insecure TLS connector (tonic's default connector already does this), and service-forward accepted sockets - supervisor: direct-tcpip connect into the sandbox netns, TCP relay target dials, egress proxy accepted connections, and all upstream CONNECT/HTTP dials (via a new connect_upstream helper) Setting TCP_NODELAY on connect is best-effort: a failure only costs latency, so we log and continue rather than fail the connection. The sandbox SSH transport rides a unix domain socket and gRPC client channels use tonic defaults (nodelay on), so no change is needed there. Fixes NVIDIA#2219 Signed-off-by: Jim Meyer <jim@meyer4hire.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every small request/response through the sandbox relay path paid a fixed
~44 ms penalty because no socket in the path disabled Nagle's algorithm, so
sub-MSS ("tinygram") writes stalled waiting on TCP delayed ACKs at each hop.
This sets
TCP_NODELAYon every latency-sensitive TCP socket in the tunneland egress-proxy path, cutting the marginal cost of a small governed
request from ~44 ms to ~0.3–0.5 ms — within a few hundred microseconds of
the host loopback floor — with no regression on bulk transfers.
Opening as a draft to invite discussion of the approach before it's marked
ready.
Related Issue
Fixes #2219
Changes
TCP_NODELAYon accepted connections on the publiclistener (gRPC relay frames and WS tunnel writes).
stream, the insecure TLS connector (matching tonic's default connector),
and service-forward accepted sockets.
target dials (
connect_tcp_target), egress-proxy accepted connections,and all upstream CONNECT/HTTP dials (via a new
connect_upstreamhelper).TCP_NODELAYon connect is best-effort: a failure only costslatency, so we log and continue rather than fail the connection.
TCP_NODELAYis set on the proxy upstream,direct-tcpip, and TCP-relay connect paths.
The sandbox SSH transport rides a unix domain socket and gRPC client
channels use tonic defaults (nodelay on), so no change is needed there.
Testing
mise run pre-commitpassesconnect hops)
with a component-isolation benchmark ladder instead (below)
A/B benchmark ladder (medians over 30 reps/cell; macOS arm64, Docker
compute driver). Tools and full results are on a temporary branch:
relay-latency-benchmarking.Keep-alive marginal cost per small request:
No regression (worst cases): 10 MB response 58.6 → 59.0 ms (unchanged);
10 MB request 90.4 → 46.0 ms (Nagle-throttled uploads got ~2× faster);
L7 body-rewrite rung 1 KB 63.7 → 14.9 ms.
Checklist
socket-option tuning with no change to subsystem boundaries or data flow