fix(atenet): propagate trace context through router and namespace span/metric attrs#428
Closed
SURAJ KUMAR (krsnaSuraj) wants to merge 1 commit into
Closed
Conversation
…n/metric attrs Fixes trace detachment in the router where two issues broke the trace chain: 1. Primary: resumer.go's singleflight used context.Background() inside DoChan, so otelgrpc saw no parent span and ateapi handler spans appeared in a separate trace. Now propagates the caller's span context into the background context. 2. Secondary: the ext_proc HeadersResponse only rewrote :authority and never injected traceparent/tracestate into the header mutation. Now injects the trace context so the upstream worker receives the same trace even when Envoy tracing is not configured. 3. Span and metric attributes used bare keys (atespace, actor, etc.) instead of the ate.* convention established in agent-substrate#412. Renamed to ate.atespace, ate.actor_name, ate.target_addr, ate.actor_template_namespace, ate.actor_template_name, ate.outcome. Fixes agent-substrate#427
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #427
Root cause
Two bugs broke the trace chain through the router, creating two separate traces instead of one connected trace for each ResumeActor request:
1. Singleflight detaches trace context (primary)
resumer.go:58 uses context.Background() inside singleflight.DoChan(). The otelgrpc client handler reads the trace context from the context passed to the gRPC call, so the ateapi handler spans ended up in a separate trace from the router's ResumeActor span.
Fix: Propagate the caller's span context into the background context via trace.ContextWithSpanContext. The caller's lifecycle detachment is preserved (background context parent), but the trace chain is maintained.
2. No traceparent injected into upstream request (secondary)
extproc.go:179-187 only rewrote the :authority header in the HeadersResponse mutation. When Envoy tracing is not configured (no OTLP collector), Envoy does not inject traceparent into upstream requests, so the worker pod receives no trace context.
Fix: After building the header mutation, call injectTraceContext(ctx, mutation) which uses otel.GetTextMapPropagator().Inject() to write traceparent/tracestate into the mutation set headers.
3. Non-namespaced attribute keys
Span and metric attributes used bare keys (atespace, actor, actor_template_namespace, etc.) instead of the ate.* namespaced convention established in #412.
Fix: Renamed attributes across extproc.go and resumer.go:
Also added missing ate.atespace and ate.actor_name attributes to the ExtProc.RequestHeaders span (previously had zero custom attributes).
Testing