EventEmitter: offAndWait()/removeAllListenersAndWait() for the destroy-vs-in-flight-handler race

[ptesavol]

Problem

An owner that removes a listener and immediately frees the state the listener captured (the ListeningRpcCommunicator::destroy() pattern behind the phase-C6 SIGSEGVs) can race a handler invocation still running on another thread — a use-after-free. The phase-C6 attempt made off() itself wait for in-flight invocations and was reverted before merge: it hung Layer1ScaleTest.MultipleLayer1Dht and ContentDeliveryLayerNodeLayer1Test.HappyPathSixtyFourNodes.

Root cause of the hang (runtime-proven with thread samples of the hung tests)

Blocking off() deadlocks in two distinct shapes, so it cannot be the default:

1. Handler↔handler cycle — one worker ran a connection's Data handler (handshake success → stopHandshaker() → off<Disconnected>) while another ran the same connection's Disconnected handler (close tie-break → off<Data>); each waited forever for the other's invocation to end. One of the two workers was a Simulator association-executor delivery task, so its inFlightOperations decrement never ran — that is exactly why the hang surfaced as Simulator::stop()'s drain never completing.
2. Caller-held-lock ABBA — ConnectionManager::acceptNewConnection holds the Endpoint state mutex while detachFromPendingConnection removes the Connected listener; the in-flight Connected handler blocks on that same mutex.

Design

• off()/removeAllListeners() keep their historical non-blocking semantics.
• New offAndWait()/removeAllListenersAndWait() remove the listener(s), cancel not-yet-started invocations of an executing emit loop, and wait until no invocation runs on another thread.
• The wait is skipped when the caller is itself inside any handler invocation, tracked by a thread_local invocation depth bumped around every handler call. A blocked waiter therefore never runs a handler itself, so no cycle of handlers waiting on each other can form (shape 1). Waiting callers must not hold locks the event's handlers take (shape 2) — documented on the methods.
• ListeningRpcCommunicator::destroy() now uses the waiting variants.
• Simulator hardening found during the investigation: folly::SerialExecutor swallows task exceptions, so a throwing delivery call-out would silently skip the inFlightOperations decrement and park stop()'s drain forever. The decrement now runs on all paths (exception logged). No such exception was observed at runtime; this closes the hazard.

Tests

• OffWaitsForInFlightHandlerBeforeOwnerFreesState / RemoveAllListenersWaitsForInFlightHandler — the destroy-vs-in-flight-handler regression tests; they fail under non-waiting semantics.
• HandlersRemovingEachOtherAcrossThreadsDoNotDeadlock — replica of deadlock shape 1; completes via the depth guard.
• Scale verification: reintroducing blocking off() deadlocked 8 s into MultipleLayer1Dht (two workers parked forever, drain starved); with this design MultipleLayer1Dht and the 64-node test passed 8/8 runs (instrumented + clean builds) with zero blocked-wait diagnostics. Full emitter suite 22/22; streamr-eventemitter and streamr-dht lint clean.

🤖 Generated with Claude Code

[ptesavol]

CI failure diagnosis: pre-existing hang, not this change — rebased onto main

The macOS-leg failure (WebrtcFullNodeNetworkTest.HappyPath, 1200 s ctest timeout) is the pre-existing joinScope teardown-drain hang, not the emitter change. Evidence:

• The CI log shows SetUp crawling ~20 min, throwing "Timed out" (the connectivity wait), and TearDown then hanging silently until the ctest kill — the same signature PR Phase D1: streamrNode* C API — the full network node in the shared library #90's investigation hit on ubuntu-latest before this branch existed.
• Reproduced locally by pinning SharedExecutors to the CI runner's pool sizes (worker=4, background=2): 6/16 runs hang on the pre-rebase binary and 1/8 on a rebased binary. In all seven captured thread samples the main thread is in NetworkStack::stop() → GuardedAsyncScope::close() → Baton::wait and there are zero offAndWait frames — the new wait paths are not involved.
• The full drain fix (cancel-before-drain + closeAsync) is still in flight from the PR Phase D1: streamrNode* C API — the full network node in the shared library #90 follow-up work; until it lands this test can still flake on starved runners at a low rate.

Rebased onto current main (post-#90) and re-verified locally: emitter suite 22/22, Layer1ScaleTest.MultipleLayer1Dht pass, 64-node pass, WebrtcFullNodeNetworkTest.HappyPath pass.