feat(payload): v1.0 (idx/ETP) trace agent payload - hardening, wire-format validation, and fingerprint corpus - #1929
Draft
ajgajg1134 wants to merge 7 commits into
Draft
ajgajg1134 wants to merge 7 commits into
ajgajg1134 wants to merge 7 commits into
Conversation
Extends the trace agent payload generator with a v1.0 variant, targeting the /v1.0/traces endpoint. The v1.0 format is a structured tracer payload (streaming string table, trace chunks, spans with attributes, links, and events), so unlike v0.4 it is generated from a user-declared service graph: a set of services, each declaring operations, each of which may call operations on other services. One payload is one tracer POST. The generator emits spans the trace-agent's normalizer leaves untouched (non-empty service/name/resource, non-zero span IDs, timestamps within signed 64-bit bounds), so a target's output can be compared against what was sent without normalization noise. Blocks are single tracer payloads whose chunk count is scaled toward the requested block size; an undersized request writes nothing and is treated as a rejected block, matching the block cache's adaptation.
…ng blocks Sizing a block to fit max_bytes probed candidate chunk counts by regenerating every chunk from scratch, redoing generation - the dominant cost - several times per block. Chunks are now generated exactly once into a pool and each probe re-encodes a prefix of it, matching the v0.4 approach. A stale last_span_count on the early-return paths and the floor*2 overflow are fixed along the way. Per-span work moves out of the generation hot path: operations are resolved once at construction into interned (Arc) service, name, resource, span_type, component, env, and version handles plus key-sorted attribute vectors, so building a span clones cheap handles instead of re-deriving sorted attributes from the configuration HashMap. The suboperations clone that existed only to appease the borrow checker is gone. Validation hardening: duplicate operation detection and suboperation reference checks now run in Config::valid (config-check sees them) via an FxHashSet, and deny_unknown_fields on Service, Operation, and SubOperation makes a mistyped field fail loudly instead of silently dropping the call graph. The entry-point selection no longer expects; it reports an error. Determinism and span-nesting invariants are now proptests over arbitrary seeds rather than single-seed unit tests.
…tures The hand-written byte-literal tests only proved the encoder agrees with itself; a wrong field number or attribute layout would pass all of them. These golden tests close that gap with two fixtures captured from a real dd-trace-go v2.11.0-dev.1 tracer (via the apm-v1-trace-smoke application posting to a recording sink): - golden_tracer_payload.bin: three services with nested parent/child spans and error spans, six chunks per payload. - golden_links_events_payload.bin: the same configuration with a span link (16-byte trace ID, attributes, flags) and a span event carrying one attribute of each scalar type. A schema-aware test decoder, transcribed from the reference encoder and decoder (dd-trace-go payload_v1.go, saluki decoders/datadog) rather than from this module's encoder, decodes the fixtures and this encoder's output into the same structures and requires them to agree. The decoder accepts both wire styles the references produce: compact and fixed-width integers, and the omit-default style this encoder uses alongside the tracer's always-present chunk fields.
Real tracers mark a failed span with the error bit plus error.type and
error.message string attributes - that is what the golden captures show
dd-trace-go sending and what the receiving agent's error tracking reads.
Error injection now matches that shape instead of attaching
http.status_code: 500 to every span regardless of span type.
INTENTIONAL fingerprint change, per the ci/fingerprint policy: the
corpus config sets error_rate 0.1, so roughly one span in ten now
carries different attributes.
trace_agent_v1: d7380752e9daf11655a97ac2548c37113d6efe0865f4a6f87d8c8e637f5fa6db
-> c9070ba10cd00c859be56cbf1600a10f475a4e547b73c85f495cb701cc1d130d
entropy: 7.0359 -> 7.0414 (increased; the constant status-code string is
replaced by two longer synthetic error attributes).
The regenerated fingerprint.txt is committed alongside the code change,
as the policy requires. Also adds ci/fingerprints/trace_agent_v1/ (the
determinism corpus for the new payload) and
examples/trace-agent-v1.yaml (a worked service-graph example,
counterpart to trace-agent-v04.yaml).
ajgajg1134
force-pushed
the
trace-agent-v1
branch
from
September 11, 2026 17:54
c1b0dee to
abc4eaf
Compare
… agent Import ValueWriteError instead of using 4-segment paths, drop the `as _` import of Serialize, and apply cargo fmt.
The golden tests validate lading's hand-rolled v1.0 encoder against real dd-trace-go captures, but carried two pieces of avoidable bulk: a hand-written base64 decoder and ~180 lines of hand-written MessagePack marker matching. Use the base64 crate as a dev-dependency (already resolved in the lockfile) and delegate primitive MessagePack framing to rmp::decode, which the encoder already depends on for rmp::encode. The schema layer stays hand-transcribed from the reference implementations -- field identifiers, the streaming string table, and the [key, type, value] attribute layout -- so the tests remain an independent check on the encoder rather than a mirror of it. Marker parsing is rmp's own well-tested concern, so sharing it creates no blind spot. Verified by mutating a span field identifier in the encoder: the two golden round-trip tests fail while all 16 inline byte-literal tests pass, confirming the coverage that matters is intact. golden.rs: 753 -> 629 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement generator support for emitting DD v1 message pack tracer payloads
Added a "known good" example from an application using dd-trace-go emitting v1 payloads.
Notes for reviewers
Checklist
ci/fmtcargo clippycargo nextest run --workspace --exclude sheepdog(504 passed)ci/fingerprintpolicy followed)lading config-check