fix(atenet): propagate trace context through router and namespace span/metric attrs by krsnaSuraj · Pull Request #429 · agent-substrate/substrate

SURAJ KUMAR (krsnaSuraj) · 2026-07-12T18:50:54Z

Fixes #427

Root cause

Two bugs broke the trace chain through the router, creating two separate traces instead of one connected trace for each ResumeActor request:

1. Singleflight detaches trace context (primary)

resumer.go:58 uses context.Background() inside singleflight.DoChan(). The otelgrpc client handler reads the trace context from the context passed to the gRPC call, so the ateapi handler spans ended up in a separate trace from the router's ResumeActor span.

Fix: Propagate the caller's span context into the background context via trace.ContextWithSpanContext. The caller's lifecycle detachment is preserved (background context parent), but the trace chain is maintained.

2. No traceparent injected into upstream request (secondary)

extproc.go:179-187 only rewrote the :authority header in the HeadersResponse mutation. When Envoy tracing is not configured (no OTLP collector), Envoy does not inject traceparent into upstream requests, so the worker pod receives no trace context.

Fix: After building the header mutation, call injectTraceContext(ctx, mutation) which uses otel.GetTextMapPropagator().Inject() to write traceparent/tracestate into the mutation set headers.

3. Non-namespaced attribute keys

Span and metric attributes used bare keys (atespace, actor, actor_template_namespace, etc.) instead of the ate.* namespaced convention established in #412.

Fix: Renamed attributes across extproc.go and resumer.go to match the dot-hierarchy convention from #412:

atespace -> ate.atespace
actor -> ate.actor.name
actor_template_namespace -> ate.actor.template.namespace
actor_template_name -> ate.actor.template.name
target_addr -> ate.target_addr
outcome -> ate.outcome

Also added missing ate.atespace and ate.actor.name attributes to the ExtProc.RequestHeaders span (previously had zero custom attributes).

Testing

Existing TestExtProcHeadersEvaluation updated to handle additional trace headers in the mutation (searches for :authority among SetHeaders instead of asserting exactly one entry).
TestHandleRequestHeadersDoesNotLogSensitiveData unaffected (no mutation assertion).
TestActorResumer_ResumeActor unaffected (tests the resumed actor, not span internals).
TestInjectTraceContext_AddsTraceparentMatchingParentSpan verifies injected traceparent matches parent span trace ID and uses OVERWRITE AppendAction.
TestInjectTraceContext_AppendActionDefaultsToOverwrite verifies AppendAction is OVERWRITE_IF_EXISTS_OR_ADD (not APPEND).
TestActorResumer_ResumeChildSharesParentTraceID verifies ResumeActor span shares parent trace ID and parent span ID.
go vet clean, all 30+ tests pass.

fix(atenet): connect router traces across ateapi, atelet, and worker by propagating span context through singleflight and injecting traceparent into ext_proc mutations; namespace all span and metric attributes to ate.*

BREAKING: The route_duration metric labels are renamed from actor_template_namespace/actor_template_name/outcome to ate.actor.template.namespace/ate.actor.template.name/ate.outcome, changing time series identity. Update dashboards and alerting rules that reference the old label names.

Bowei Du (bowei) · 2026-07-13T06:18:25Z

Max Smythe (@maxsmythe) can you take a look?

Krisztian F (krisztianfekete)

Added some comments!

Krisztian F (krisztianfekete) · 2026-07-13T08:47:38Z

+	headers := make(map[string]string)
+	otel.GetTextMapPropagator().Inject(ctx, propagation.MapCarrier(headers))
+	for k, v := range headers {
+		mutation.SetHeaders = append(mutation.SetHeaders, &corev3.HeaderValueOption{


You're not setting AppendAction, so this defaults to APPEND_IF_EXISTS_OR_ADD. The request already has a traceparent, so Envoy appends a second value and not replace it. The w3c propagator receives a multi value traceparent, which is invalid per spec, so the worker starts a fresh root span.

Great catch! I''ve set AppendAction to OVERWRITE_IF_EXISTS_OR_ADD on all injected trace headers so Envoy replaces the existing traceparent instead of appending a second (invalid) value.

Krisztian F (krisztianfekete) · 2026-07-13T08:49:09Z

-			headerOption := mutation.GetSetHeaders()[0]
-			if strings.ToLower(headerOption.Header.Key) != ":authority" {
-				t.Errorf("invalid resulting dynamic parameter key: %s", headerOption.Header.Key)
+			headers := mutation.GetSetHeaders()


Could you add a small test with a local TracerProvider + in-memory exporter, start a parent span, and assert the mutation has a traceparent matching it? Also same idea for the resumer reparenting.

Done! Added TestInjectTraceContext_AddsTraceparentMatchingParentSpan - sets up a local TracerProvider with in-memory exporter, starts a parent span, calls injectTraceContext, and asserts the injected traceparent matches the parent span''s trace ID. Also added TestInjectTraceContext_AppendActionDefaultsToOverwrite as a focused AppendAction check.

Krisztian F (krisztianfekete) · 2026-07-13T08:50:15Z

 		defer bgCancel()
+		// Propagate the caller's span context so the gRPC spans are children
+		// of ResumeActor rather than appearing as a separate trace.
+		bgCtx = trace.ContextWithSpanContext(bgCtx, trace.SpanContextFromContext(ctx))


This looks good, but can you please add test asserting the resume child span shares the parent traceid?

Done! Added TestActorResumer_ResumeChildSharesParentTraceID - sets up a local TracerProvider + in-memory exporter, starts a parent span, calls ResumeActor, and asserts the ResumeActor span shares the parent''s trace ID and parent span ID.

Krisztian F (krisztianfekete) · 2026-07-13T08:51:34Z


-	// Route by rewriting the :authority header.
+	span.SetAttributes(
+		attribute.String("ate.target_addr", targetAddr),


This is exactly what the stable OTel server.address + server.port semconv attrs are for. Can you please use those?

Good idea - much cleaner than a custom attribute. Replaced attribute.String(''ate.target_addr'', targetAddr) with semconv.ServerAddress(workerIP) + semconv.ServerPort(80) so we use the stable OTel semconv conventions.

Krisztian F (krisztianfekete) · 2026-07-13T08:52:43Z

+			attribute.String("ate.atespace", atespace),
+			attribute.String("ate.actor.name", actorName),


FYI these are what I am setting in #412 Can we plan the rebase to call the ateattr helpers instead of redoing it once #412 is merged?

Good call! Once #412 merges, I''ll rebase this branch and replace the inline ate.* attribute strings with the ateattr helpers. Will track that as a follow-up so it doesn''t block the trace context fix here.

Krisztian F (krisztianfekete) · 2026-07-13T08:54:47Z

+		attribute.String("ate.actor.template.namespace", tmplNs),
+		attribute.String("ate.actor.template.name", tmplName),
+		attribute.String("ate.outcome", outcome),


Renaming these metric labels changes the time series identity, I left these out of #412 for this reason. Totally fine to do it, but make sure to call it out explicitly in the release note as it's a "breaking change".

Good point - completely agreed. I''ve updated the release note in the PR description to explicitly call out the route_duration label rename as BREAKING, noting that dashboards and alert rules need updating.

…n/metric attrs Two bugs broke the trace chain through the router, creating two separate traces instead of one connected trace for each ResumeActor request. 1. Singleflight detaches trace context (primary): resumer.go used context.Background() inside singleflight.DoChan(). Fixed by propagating the caller's span context into the background context via trace.ContextWithSpanContext. 2. No traceparent injected into upstream request (secondary): extproc.go only rewrote the :authority header. Fixed by calling injectTraceContext(ctx, mutation) which uses otel.GetTextMapPropagator() to write traceparent/tracestate with OVERWRITE_IF_EXISTS_OR_ADD AppendAction. 3. Non-namespaced attribute keys: Renamed span and metric attributes to ate.* convention (agent-substrate#412 style). Replaced custom target_addr attribute with stable OTel semconv server.address + server.port.

SURAJ KUMAR (krsnaSuraj) · 2026-07-15T07:21:39Z

Krisztian F (@krisztianfekete) Thanks for the thorough review! I've addressed all the feedback and pushed the changes. Mind taking another look when you get a chance?

SURAJ KUMAR (krsnaSuraj) mentioned this pull request Jul 12, 2026

atenet-router traces are detached + do not use ate.* attribute keys #427

Open

SURAJ KUMAR (krsnaSuraj) force-pushed the fix/atenet-trace-context-propagation branch 3 times, most recently from 89b4d22 to 4c6bd93 Compare July 12, 2026 19:13

Bowei Du (bowei) requested a review from Max Smythe (maxsmythe) July 13, 2026 06:18

Krisztian F (krisztianfekete) suggested changes Jul 13, 2026

View reviewed changes

SURAJ KUMAR (krsnaSuraj) force-pushed the fix/atenet-trace-context-propagation branch from 900bcca to 76a2de8 Compare July 15, 2026 07:16

SURAJ KUMAR (krsnaSuraj) force-pushed the fix/atenet-trace-context-propagation branch from 76a2de8 to d9b367e Compare July 15, 2026 07:17

		attribute.String("ate.atespace", atespace),
		attribute.String("ate.actor.name", actorName),

Uh oh!

Conversation

SURAJ KUMAR (krsnaSuraj) commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

1. Singleflight detaches trace context (primary)

2. No traceparent injected into upstream request (secondary)

3. Non-namespaced attribute keys

Testing

Uh oh!

Bowei Du (bowei) commented Jul 13, 2026

Uh oh!

Krisztian F (krisztianfekete) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SURAJ KUMAR (krsnaSuraj) commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SURAJ KUMAR (krsnaSuraj) commented Jul 12, 2026 •

edited

Loading