design(platform): compass-server + runner OTel emission and end-to-end trace continuity (RIG-2685) by rigel-mintaka · Pull Request #690 · RigelBuild/compass

rigel-mintaka · 2026-08-27T22:19:57Z

Sibling record of the frozen observability-architecture design (#656 §T4b), covering the server + runner Go OTel emission leg and end-to-end W3C traceparent trace continuity across the server → runner → agent boundary. Builds on the frozen #649 propagation contract (server is origin, runner relays, agent continues). Ledger-impact: none (the platform bucket is ungoverned).

Approach — two legs, one record

Emission: one go/internal/otel bootstrap package (the only TracerProvider construction site), OTLP/http-protobuf exporter, off by default and endpoint-gated on OTEL_EXPORTER_OTLP_ENDPOINT, wired into both run()s after slog.SetDefault and before Serve/Run. RPC spans come free from connectrpc.com/otelconnect interceptors on the existing chains.
Propagation: server-origin stamping at control-wrap (deliverOp/steerOp gain a traceparent param), runner relays the op verbatim, agent continues. The one genuinely new seam is the in-process events bus — Bus.Publish takes no ctx, so the trace context dies between PostMessage and the delivery consumer; bridged by a narrow PublishCtx(ctx, payload) used only by the message-origin publisher, stamping the traceparent onto the existing Stamped[P] envelope.
Response boundary: a traceresponse response-header interceptor makes the server-minted trace id readable by the UI/PostHog correlation seam (docs(platform): observability & in-product data architecture design record #656 T6/RIG-2874).

Ownership

compass-obs owns the emission + stamping + relay-span + e2e-test tasks (T1/T2/T3/T5/T6). The agent.proto traceparent field (T4) is compass-server's file zone — a stacked seam that lands first and unblocks compass-agent's decode (RIG-2871). compass-native owns the image rebuild the new Go deps force.

Open Questions for the freeze gate

Metrics scope — recommend traces-only now, metrics a documented follow-up.
traceparent on the public SessionInjection observation — recommend defer (public-API surface, core does not need it; SEA-1267 gen-fence).
Bus bridge shape — recommend the PublishCtx split over a full signature cutover.
traceresponse header name — recommend the W3C draft name.
Origin span for agent-authored posts (red-team finding) — recommend a fresh per-call span in the hub's post arm; hard constraint that the Sessions-stream span never leaks into the publish ctx.

Red-teamed by a design-critic pass; all mechanical findings folded and the two coverage-gap forks (held-deliver continuity, agent-post origin) surfaced as load-bearing Open Questions.

Spec-impact: none. Refs RIG-2685

Co-authored-by: Matt Wilkinson matt@rigel.build

linear-code · 2026-08-27T22:20:01Z

RIG-2685

github-actions · 2026-08-27T22:21:35Z

Compass engineering docs preview: https://compass-obs-rig-2685-server.compass-eng-docs.pages.dev

Deployed from compass-obs/rig-2685-server-runner-otel-design at ab10025.

Changed pages:

docs/designs/platform/compass-server-runner-otel/design.md

…d trace continuity (RIG-2685) Sibling record of the frozen observability-architecture design (#656 §T4b), covering the server + runner Go OTel emission leg and end-to-end W3C `traceparent` trace continuity across the server → runner → agent boundary. Builds on the frozen #649 propagation contract (server is origin, runner relays, agent continues). `Ledger-impact: none` (the `platform` bucket is ungoverned). ### Approach — two legs, one record - **Emission:** one `go/internal/otel` bootstrap package (the only `TracerProvider` construction site), OTLP/http-protobuf exporter, off by default and endpoint-gated on `OTEL_EXPORTER_OTLP_ENDPOINT`, wired into both `run()`s after `slog.SetDefault` and before `Serve`/`Run`. RPC spans come free from `connectrpc.com/otelconnect` interceptors on the existing chains. - **Propagation:** server-origin stamping at control-wrap (`deliverOp`/`steerOp` gain a `traceparent` param), runner relays the op verbatim, agent continues. The one genuinely new seam is the in-process events bus — `Bus.Publish` takes no ctx, so the trace context dies between `PostMessage` and the delivery consumer; bridged by a narrow `PublishCtx(ctx, payload)` used only by the message-origin publisher, stamping the traceparent onto the existing `Stamped[P]` envelope. - **Response boundary:** a `traceresponse` response-header interceptor makes the server-minted trace id readable by the UI/PostHog correlation seam (#656 T6/RIG-2874). ### Ownership compass-obs owns the emission + stamping + relay-span + e2e-test tasks (T1/T2/T3/T5/T6). The `agent.proto` `traceparent` field (T4) is compass-server's file zone — a stacked seam that lands first and unblocks compass-agent's decode (RIG-2871). compass-native owns the image rebuild the new Go deps force. ### Open Questions for the freeze gate 1. Metrics scope — recommend traces-only now, metrics a documented follow-up. 2. `traceparent` on the public `SessionInjection` observation — recommend defer (public-API surface, core does not need it; SEA-1267 gen-fence). 3. Bus bridge shape — recommend the `PublishCtx` split over a full signature cutover. 4. `traceresponse` header name — recommend the W3C draft name. 5. Origin span for agent-authored posts (red-team finding) — recommend a fresh per-call span in the hub's post arm; hard constraint that the Sessions-stream span never leaks into the publish ctx. Red-teamed by a design-critic pass; all mechanical findings folded and the two coverage-gap forks (held-deliver continuity, agent-post origin) surfaced as load-bearing Open Questions. Spec-impact: none. Refs RIG-2685 Co-authored-by: Matt Wilkinson <matt@rigel.build>

T1 of the frozen server+runner OTel design record (`docs/designs/platform/compass-server-runner-otel/design.md`, RIG-2685, #690). Establishes the single Go OTel convention for the Compass backend — there was no Go OTel instrumentation before this; the server and runner emission wiring (T2/T3) and the propagation leg (T5) all build on it. New package `go/internal/otel`: - `Config{ServiceName, ServiceVersion, Endpoint}` + `SetupTracerProvider`/`SetupMeterProvider(ctx, cfg) (shutdown, err)` — off by default, endpoint-gated on `OTEL_EXPORTER_OTLP_ENDPOINT`: empty endpoint installs no global provider and returns a non-nil no-op shutdown (zero overhead, no network egress), mirroring the agent's `src/transport/otel-layer.ts` posture. Non-empty installs global providers whose resource carries `service.name`/`service.version`, OTLP-over-HTTP/protobuf exporters that read the endpoint env themselves (standard per-signal `/v1/traces` vs `/v1/metrics` suffixing), and the global W3C `TraceContext` propagator. - `Traceparent(ctx)` / `ContextWithTraceparent(ctx, tp)` — thin wrappers over the shared W3C propagator; empty/no-span yields `""`, malformed input returns the context unchanged. - `NewTraceResponseInterceptor()` — a connect unary interceptor that sets a `traceresponse` response header (`00-<traceid>-<spanid>-<flags>`) from the active span, the UI/PostHog trace-id correlation source; no-op when no active span. OTel Go SDK deps added at v1.46.0 (above the ≥v1.23.0 floor the record sets for `Span.AddLink`, used later by T5). ### FOD vendorHash lockstep The new Go deps move the fetched Go module set, invalidating the pinned `vendorHash` the two `buildGoModule` FODs share (`flake.nix` and `guest-image/default.nix`, both documented to carry the same value since guestd's `proxyVendor` fetches the full module graph). Left stale, the `compass-guest-image` and `flake-gate:flake-check` builds fail `hash mismatch in fixed-output derivation …-go-modules.drv`. Per the repo's gomod↔FOD lockstep convention (the same refresh Renovate's `refresh-fod-hashes.ts` rides in-PR on a dep bump), both pins are refreshed here to the SRI the build reports, so a gomod change and its FOD refresh land in one commit and main never goes red between merges. Verified locally: `moon run compass-guest-image:build flake-gate:flake-check` both pass. Built, tested (8/8), vetted, gofmt- and golangci-lint-clean. Spec-impact: none. Refs RIG-2888, RIG-2685, RIG-2895. Co-authored-by: Matt Wilkinson <matt@rigel.build>

* feat(otel): add go/internal/otel bootstrap package (RIG-2888) T1 of the frozen server+runner OTel design record (`docs/designs/platform/compass-server-runner-otel/design.md`, RIG-2685, #690). Establishes the single Go OTel convention for the Compass backend — there was no Go OTel instrumentation before this; the server and runner emission wiring (T2/T3) and the propagation leg (T5) all build on it. New package `go/internal/otel`: - `Config{ServiceName, ServiceVersion, Endpoint}` + `SetupTracerProvider`/`SetupMeterProvider(ctx, cfg) (shutdown, err)` — off by default, endpoint-gated on `OTEL_EXPORTER_OTLP_ENDPOINT`: empty endpoint installs no global provider and returns a non-nil no-op shutdown (zero overhead, no network egress), mirroring the agent's `src/transport/otel-layer.ts` posture. Non-empty installs global providers whose resource carries `service.name`/`service.version`, OTLP-over-HTTP/protobuf exporters that read the endpoint env themselves (standard per-signal `/v1/traces` vs `/v1/metrics` suffixing), and the global W3C `TraceContext` propagator. - `Traceparent(ctx)` / `ContextWithTraceparent(ctx, tp)` — thin wrappers over the shared W3C propagator; empty/no-span yields `""`, malformed input returns the context unchanged. - `NewTraceResponseInterceptor()` — a connect unary interceptor that sets a `traceresponse` response header (`00-<traceid>-<spanid>-<flags>`) from the active span, the UI/PostHog trace-id correlation source; no-op when no active span. OTel Go SDK deps added at v1.46.0 (above the ≥v1.23.0 floor the record sets for `Span.AddLink`, used later by T5). ### FOD vendorHash lockstep The new Go deps move the fetched Go module set, invalidating the pinned `vendorHash` the two `buildGoModule` FODs share (`flake.nix` and `guest-image/default.nix`, both documented to carry the same value since guestd's `proxyVendor` fetches the full module graph). Left stale, the `compass-guest-image` and `flake-gate:flake-check` builds fail `hash mismatch in fixed-output derivation …-go-modules.drv`. Per the repo's gomod↔FOD lockstep convention (the same refresh Renovate's `refresh-fod-hashes.ts` rides in-PR on a dep bump), both pins are refreshed here to the SRI the build reports, so a gomod change and its FOD refresh land in one commit and main never goes red between merges. Verified locally: `moon run compass-guest-image:build flake-gate:flake-check` both pass. Built, tested (8/8), vetted, gofmt- and golangci-lint-clean. Spec-impact: none. Refs RIG-2888, RIG-2685, RIG-2895. Co-authored-by: Matt Wilkinson <matt@rigel.build> * fix(otel): address T1 review — wrap bootstrap errors, add meter + interceptor tests (RIG-2888) Review-fix commit atop the T1 bootstrap PR (#699). Addresses the low-severity findings from the review pass; the single medium (exporter-endpoint knob reconciliation) is a design decision routed separately and does not change T1 code. - **provider.go:** wrap the three bootstrap errors (resource build, trace exporter, metric exporter) with `fmt.Errorf("otel: ...: %w", err)` so a fatal startup log names the failing stage (house Go idiom). - **otel_test.go:** strengthen `TestSetupMeterProviderDisabled` with a sentinel assertion that no SDK MeterProvider is installed on the disabled path (mirrors the tracer test); add `TestSetupMeterProviderEnabled`; add `TestFormatTraceResponse` and `TestTraceResponseInterceptor` (valid-span sets header, no-span sets none, handler error passthrough) — the interceptor is a cross-task contract symbol and was previously uncovered in-package. Refs RIG-2888. Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <matt@rigel.build>

rigel-mintaka force-pushed the compass-obs/rig-2685-server-runner-otel-design branch from b7b8603 to 1f371b3 Compare August 27, 2026 23:10

mattwilkinsonn marked this pull request as ready for review August 27, 2026 23:21

mattwilkinsonn self-requested a review as a code owner August 27, 2026 23:21

rigel-mintaka force-pushed the compass-obs/rig-2685-server-runner-otel-design branch from 1f371b3 to ab10025 Compare August 27, 2026 23:56

mattwilkinsonn approved these changes Aug 28, 2026

View reviewed changes

mattwilkinsonn merged commit 4efc3b1 into main Aug 28, 2026
13 checks passed

mattwilkinsonn deleted the compass-obs/rig-2685-server-runner-otel-design branch August 28, 2026 01:38

rigel-mintaka mentioned this pull request Aug 28, 2026

feat(otel): add go/internal/otel bootstrap package (RIG-2888) #699

Merged

Navigation Menu

Uh oh!

design(platform): compass-server + runner OTel emission and end-to-end trace continuity (RIG-2685) - #690

design(platform): compass-server + runner OTel emission and end-to-end trace continuity (RIG-2685)#690
mattwilkinsonn merged 1 commit into
mainfrom
compass-obs/rig-2685-server-runner-otel-design

rigel-mintaka commented Aug 27, 2026

Uh oh!

linear-code Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rigel-mintaka commented Aug 27, 2026

Approach — two legs, one record

Ownership

Open Questions for the freeze gate

Uh oh!

linear-code Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Aug 27, 2026 •

edited

Loading