Fix the full-node teardown hang: cancel and suspend the joinScope drain

[ptesavol]

Problem

WebrtcFullNodeNetworkTest.HappyPath hit the 1200 s ctest timeout twice in a row on the ubuntu-latest leg of #90 (run 29248801288, attempt 1): one of the eight concurrent NetworkStack::start() calls threw folly's FutureTimeout ("Timed out") from waitForNetworkConnectivity at exactly +10 s, and the process then produced no output for ~20 minutes until ctest killed it. The same commit passed on the macos-26 and linux-arm64 legs.

Root cause (runtime-evidence diagnosis)

By the time the exception fires, every node has already detached a fire-and-forget joinDht task into NetworkStack's GuardedAsyncScope. In TearDown, all nine stop() coroutines run on one blockingWait drive thread, and the first stop():

1. never cancels the detached join (violating the scope's documented cancel-before-close contract — the DhtNode abort happens later in the stop sequence and cannot be reordered past the ContentDeliveryManager teardown-ordering constraints), and
2. drains it via the thread-blocking joinScope.close(), freezing every sibling stop behind it.

The drain therefore waits for the join to complete naturally, which needs shared-worker-pool capacity — every coroutine resumption, including folly::coro::timeout resumptions, needs a pool slot. The 8-node start storm starved the 4-thread CI pool (the CI log shows zero websocket accepts for the whole 10 s window), so the drain never returned. Saturating SharedExecutors::worker() locally reproduced the identical silent hang, and the queued joinDht started the same millisecond a pool thread freed — confirming the dependency.

Fix

• NetworkStack: a CancellationSource wraps the detached joinDht; stop() requests cancellation before draining, and drains via the new suspending GuardedAsyncScope::closeAsync() so concurrent sibling stops proceed meanwhile.
• GuardedAsyncScope: new closeAsync() (suspends instead of blocking; sync close() kept for destructors).
• DiscoverySession / RingDiscoverySession: merge the ambient cancellation token with the node's abort signal instead of replacing it — co_withCancellation replaces the ambient token, which made stop()-time cancellation invisible inside the sessions' RPC awaits.
• PeerDiscovery: a join cancelled by its owner's stop() no longer schedules a detached rejoin.
• CoroutineHelper: co_current_cancellation_token / cancellation_token_merge shims (CPOs, same pattern as the existing blockingWait shim).

Verification

• Instrumented before/after under the injected CI failure shape (one node's connectivity wait times out while siblings are mid-start): pre-fix the drains ran strictly serially and waited for natural join completion; post-fix all nine drains enter in the same millisecond, cancelled joins collapse in ~1.5 s, failed-SetUp teardown completes in 2.5 s.
• Under a 20 s artificial worker-pool saturation (the CI-shape wedge): pre-fix teardown stayed wedged for the entire window; post-fix it completes ~5 s after the first pool liveness.
• GuardedAsyncScopeTest adds the deadlock-shape regression test (drain that only completes if a sibling stop can progress — hangs with blocking close()).
• dht end-to-end join suite (7 tests) passes with the cancellation plumbing under heavy machine load; utils and discovery unit tests pass.

Distinct follow-up (pre-existing, not addressed here): the entry point accepting zero websocket connections for the whole 10 s window is the known accept-path degradation that made SetUp fail in the first place.

🤖 Generated with Claude Code

[ptesavol]

Completed the teardown-hang fix: four more defects diagnosed and fixed on this branch

The original joinScope fix was necessary but not sufficient — this PR's own CI run still timed out on both full-node tests (ubuntu + arm64) and segfaulted on macOS. Reproducing under a CI-emulated worker pool (SharedExecutors pinned to 4 threads, the 3-core runner size) showed 12 of 16 local runs failing, each layer hiding the next. All diagnosed with thread samples / crash reports before fixing; the branch is rebased onto main (picks up #93) and now carries six commits:

1. ContentDeliveryManager (ceac17da): destroy()'s blocking joinScope.close() drained un-cancelled startLayersAndJoinDht/handleEntryPointLeave tasks — the same mechanism this PR fixed in NetworkStack, one layer down (sampled at ContentDeliveryManager.cppm:209). Same fix: cancellation source + closeAsync().
2. StoreManager (149b2fef): detached onContactAdded replications launched with a bare .start(); they pinned the StoreManager but sent through the DhtNode's communicator, which died first (crash reports: notify<ReplicateDataRequest> resume, 13/16 once teardown got faster). Now scoped, cancelled, and drained in destroy() while the communicator is alive.
3. ConnectionLocker (520ed7f2): lockConnection/unlockConnection blockingWait'ed their RPC round-trip on pool threads — with ≥poolsize concurrent joins inside lockConnection, every worker waits for a response only those workers could process (sampled: all four StreamrWorkers wedged). Interface and implementation are now coroutines; ProxyClient keeps its synchronous behavior by wrapping at its call sites (async-ifying that path is a follow-up).
4. DhtNode (7a3a72ab): the server-RPC dispatch scope drained only at member destruction, after later-declared members were gone (crash reports: getClosestPeers dispatch resuming into freed memory). stop() now removes transport listeners with offAndWait() and drains the communicator last; the RPC APIs gate scope-adds after a drain.
5. Websocket connectors (03a7f230): both destroy()s held the connector mutex across PendingConnection::close(true) call-outs — ABBA against rtc-callback/pool threads holding a connection mutex (sampled twice). Call-outs now run outside the lock.
6. d79c601b: clang-format convergence (checker's llvm-22 binary); mostly reverts cosmetic reflow — review with whitespace hidden.

Validation (CI-emulated 4-thread pool, WebrtcFullNodeNetworkTest + WebsocketFullNodeNetworkTest): baseline this branch 12/16 failing → 15/16 passing, zero hangs. At normal pool sizes: proto-rpc suite 24/24, ConnectionLockingTest 2/2, both full-node tests 2/2 each, 64-node test 41 s, Layer1ScaleTest.MultipleLayer1Dht 326 s. Lint green on all three touched packages.

The one residual failure (1/16 under pathological load) is a distinct pre-existing UAF (neighbordiscovery::Handshaker::handshakeWithInterleaving resuming into a destroyed owner) — filed as a follow-up task chip together with an AbortableTimers lifetime bug spotted in the same runs.

🤖 Generated with Claude Code