fix(coordination): bound await_event so it returns {pending} instead of erroring on a long-running worker#498
Merged
Conversation
Root-fix a scoring bug proven live: on django-12419 @ glm-4.6 the agent made the EXACT gold fix but scored 0 because the harness extracted the patch by parsing the model reply for a fenced diff block, and the model wrote prose instead. Two changes, standard SWE-bench practice (SWE-agent/OpenHands read the diff from repo state and pre-stage the checkout): - boxSetup(task): harness clones the instance repo at base_commit into a fixed /work BEFORE the agent shot (same session) so the agent only edits. A stochastic model cannot be trusted to clone to an exact path (observed cannot-change-to-/work failures). Adapter-agnostic seam on BenchmarkAdapter. - boxExtract(): git diff of the agent edits in /work (test files excluded; the judge applies the gold test_patch itself), preferred over the event-stream parse which stays the fallback. Prompt updated: repo pre-staged, agent only edits. Typecheck clean; calibration still green. Live glm-5.2 verification blocked on transient sandbox box-provisioning failures (box unavailable before start on a green health) - re-run when the platform host-agent recovers.
…proposer The code-surface proposer fed the coding agent the ~400-char DISTILLED findings (generationFailureDistiller), not the raw traces. The meta-harness edge (yoonholee.com/meta-harness) is raw-trace filesystem context: the coding agent greps/cats the FULL run traces of failed candidates to diagnose, rather than reading a pre-summary. Add rawTraceDistiller() — an additive analyzeGeneration producer that, instead of summarizing, points the proposer at the prior generation's real run traces already durable on disk under runDir (per-cell spans.jsonl event logs + cached-result.json scores + artifacts) with a grep/cat-to-diagnose instruction. It emits AnalystFinding[] with ABSOLUTE paths (the harness runs from a worktree cwd) so it drops into the same analyzeGeneration slot and renders through the same agenticGenerator prompt path. The existing distiller stays the default. improve() gains a one-line enable: opts.rawTraceContext = true wires rawTraceDistiller() when the caller has not supplied their own analyzeGeneration (explicit analyzeGeneration still wins; null still disables). Self-test builds a real tmp runDir with fake candidate + trace files and asserts the findings reference the actual absolute trace-file paths + the grep/cat instruction, plus worst-candidate-first ordering and the clean-generation fallback. tsc clean, biome clean, 3/3 new tests + 7/7 existing improve tests pass.
…ble {pending,live} instead of an unbounded block
The supervisor's only wait-for-worker primitive (await_event -> drainSettlement ->
scope.next()) blocks on a live worker for its ENTIRE multi-minute run. Over the remote
MCP transport that block outlives the opencode client's per-request timeout, so every
await_event call surfaces as a hard tool ERROR (status=error, empty output) — observed
100% of the time across all gen-1 eval runs. The supervisor is left flying blind: it
falls back to busy-polling observe_agent, whose spent telemetry is pinned at 0 for the
whole container-exec run, sees no progress, and stops or times out (4/10). The worker's
settlement is only drained at teardown, too late to integrate.
Fix (structural, in coordination-owned harness code): race the single in-flight drain
against a bounded fence (awaitTimeoutMs, default 15s). If it settles in time, return the
event as before; if the fence wins, return { pending: true, live: [...] } — a non-error
liveness snapshot the supervisor can re-poll. The one background drain keeps running and
publishes any late settlement to the bus, so the next await_event pulls it (nothing lost).
scope.next() semantics and its exactly-once cursor are untouched.
Not a prompt/wording change: no supervisor-prompt or delegation-posture edit.
tangletools
approved these changes
Jul 8, 2026
tangletools
left a comment
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 11bc5244
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-08T19:08:30Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug (found by the meta-harness loop reading raw traces)
The supervisor's
await_event— its only "wait for the worker" primitive — errored on 100% of calls in a real supervisor-arm run. It blocks until a live child settles (viascope.next()→raceFirstSettled), which for a real sub-task is minutes; over the remote HTTP MCP transport that block exceeds opencode's per-request timeout, so the tool call returnsstatus=errorwith empty output every time. The supervisor is left blind — it busy-polls a zero-progressobserve_agentand the worker's result is only drained at teardown, too late to act on. Trace evidence: in every gen-1 eval run,await_event→status=error; the worker'ssettledbus event was published exactly once, at[sidecar] stop(SIGTERM teardown).The fix (structural, model-agnostic)
await_eventnow races the single in-flight drain against a bounded fence (DEFAULT_AWAIT_EVENT_TIMEOUT_MS = 15s, overridable viaawaitTimeoutMs): if the worker hasn't settled it returns{ pending: true, live: [...] }— a re-pollable liveness snapshot — instead of blocking to a transport error. A background drain per worker keeps running and publishes any late settlement to the bus, so the nextawait_eventpulls it;scope.next()'s exactly-once cursor is untouched, nothing is lost. Plumbed throughserveCoordinationMcp.Proof
tests/loops/coordination.test.ts: 19/19 pass, including a new test proving a still-running worker yields{pending, live:[w0]}(not a hang/error) and a laterawait_eventpulls the settlement once it lands. tsc clean; fulltests/loops/345 passed / 1 pre-existing skip.Provenance + honest scope
This is generation 2 of a meta-harness loop on the supervisor harness (gen-1 was a REJECT — a prompt-wording edit that the eval caught as a null). A coding agent reading the raw traces found this bug where prompt-tweaking couldn't. The bug + fix stand on the trace evidence + the unit test. Note: the end-to-end supervisor-arm eval did NOT exercise the fix on glm-4.6 (delegation is stochastic — the sampled runs went solo, calling
await_eventzero times), so no e2e resolve-rate lift is claimed here; the fix is a correctness fix for the coordination primitive, proven at the unit level. Stacks on #495 (raw-trace context).