[linj-glitch]

Summary

Make the request traces the fleets write self-describing for SFT export, so what Zhaoran's GLM-5.3 kernel-trace audit (2026-10-06) had to reconstruct by forensic replay is recorded at capture time. Everything is additive; with the new env vars unset, serving behaves byte-for-byte as before (one exception called out under "Behaviour changes").

Branch: linj/trace-fidelity off dev-junyix-feat-kimi-k3-serving @ e061773; merges cleanly onto the current tip (the three Helix/tripwire commits touch disjoint files).

What a reader gets

Request line (requests-<pid>.jsonl)

• thread — client_metadata.{session_id, thread_id, parent_thread_id, turn_id, root_turn_id} plus agent name / request kind from x-codex-turn-metadata. Group conversations on thread.thread_id: session is the router key, and a Codex sub-agent shares its parent's prompt_cache_key (the audit's F01). conversation_id.py's comment now says so.
• kf — the x-kf-* campaign headers without the prefix; gateway — X-Gateway-Request-Id / X-Gateway-Attempt (new, see gateway.py); sampling as sent (an absent max_output_tokens stays absent); body_sha256; build_id.

Response line (responses-<pid>.jsonl)

• prompt_tokens = {count, sha256, encoding: "int32le", source} of the token ids the engine was given — sha256(np.asarray(ids, "<i4").tobytes()). Aggregated: from the engine promise. Disaggregated: from the context worker's response in on_ctx_resp. TRTLLM_REQUEST_TRACE_PROMPT_TOKEN_IDS=1 stores the ids too. An export re-renders the body, tokenizes, and must reproduce the sha; that is the oracle the smoke ran (below).
• input_replay — what rendering changed vs the body (ReplayLedger in responses_utils: namespaces restored, bare names replayed, reasoning decoded from encrypted, orphan tool results, unknown items, tool_calls_folded, kimi_k3 flags). Rides ResponsesRequest._input_replay (PrivateAttr) and, across the ctx→proxy hop, ResponsesResponse.input_replay, stripped with the handoff.
• response_id, usage (five counts), max_output_tokens as submitted (request's or server default), output_call_ids, output_item_types; finish_reasons on chat replies; build_id. Handler-rejected requests now leave a response line too (_traced_error_response, same vocabulary as the proxy's _trace_failure; server faults answered with 400 are traced as error, not rejected_400).

Raw output (raw-<pid>.jsonl, now beside the traces when server.capture is on)

• generated token_ids, num_output_tokens, the engine's own finish_reason (threaded through PostProcessorHookChunk.finish_reason), build_id, and cheap flags: marker_literals (<|...|> spellings with counts), repetition (suffix repeats + identical-line runs, texts ≥ 2000 chars), reasoning_dots_only. Signals for an export, not verdicts. Join key: disagg_request_id (disaggregated), else engine request id == trace client_id.

Fix at source (per audit F02/F04/F06, appendix)

• TLLM_RESPONSES_DEFAULT_MAX_OUTPUT_TOKENS=<n>: default output cap for Responses requests that set none (Codex never does; the F04 loops ran uncapped). Applied where the request is rendered (workers; server.env reaches every process), echoed as the response's max_output_tokens. Off unless set.
• Parser fidelity events per call_id, in ToolCallItem.normalizations / normalize-<pid>.jsonl: name_unresolved (call to an undeclared tool); GLM argument_coerced (literal changed: 1e3→1000.0, True→true, 'a'→"a"; whitespace outside strings ignored), argument_unparsed (text delivered JSON-quoted under a non-string schema), argument_repeated. Delivered arguments are byte-identical to before.
• gateway.py: forwards X-Gateway-Request-Id / X-Gateway-Attempt to the backend (a retry's records join to one gateway line), strips client copies of the three X-Gateway-* headers, and SSE_ROTATED carries "source": "gateway", "code": "backend_rotated" so a reader can tell a gateway-injected terminal from the model's.
• serve.sh: build_id = py_manifest_sha256[:12] (only when 12 hex digits) and TRTLLM_PROVENANCE_FILE exported to every worker; provenance.txt copied beside the traces; TRTLLM_RAW_OUTPUT_DIR = the request-trace directory when capture is on. FLEET.md gains a section on what an attempt leaves under trace.request_root.

Behaviour changes to know about

1. Disaggregated Responses: the context worker's prompt ids now reach the proxy. Found by the 1P1D smoke: with num_postprocess_workers > 0 the context-only Responses reply set prompt_token_ids inside the postprocessor, where the result object has none, so the proxy relayed nothing — the generation worker rendered the prompt a second time and no digest could be recorded. OpenAIServer._attach_ctx_prompt_ids now copies them from the promise, exactly as _create_chat_response and the completions path already did. Net effect on the fleet: the generation worker consumes the relayed prompt_token_ids_b64 for Responses requests (the path that already existed for it) instead of re-rendering. Exercised end to end on the smoke (two Codex sessions, 20 calls, all completed, 22/22 digests, re-render oracle 20/20).
2. _drop_ctx_handoff now also nulls prompt_token_ids / prompt_token_ids_b64 on a context-only reply that is not handed off (EOS as first token), which used to reach the client (and the trace body) with the full id list.
3. ResponsesResponse.input_replay is dumped as null on every client-facing response and SSE snapshot, like the existing prompt_token_ids fields (~20 bytes; SDKs ignore unknown keys).
4. /v1/completions is still untraced in the aggregated server; the docstring now says so.

Env knobs (all optional; unset = previous behaviour)

Variable	Effect
TRTLLM_BUILD_ID	stamped on every record (serve.sh sets it from provenance)
TRTLLM_PROVENANCE_FILE	copied into the writer sidecar
TRTLLM_REQUEST_TRACE_PROMPT_TOKEN_IDS=1	store prompt ids beside their digest
TLLM_RESPONSES_DEFAULT_MAX_OUTPUT_TOKENS=<n>	default output cap for Responses requests without one

For the data-generation fleets: leave the cap unset. Capture is additive on purpose -- a loop is marked at export time (raw finish_reason, token counts, the repetition flag) rather than cut at serving time, because a cap set by us truncates real outputs (the A/B run saw GLM-5.2 complete 40-70k-token calls). The knob exists for deployments that want a ceiling; it is not part of the trace-quality fix. Leave the ids off unless the export wants them on disk (the sha verifies a re-render).

Verification

• Unit suites touched + 5 new files (test_request_trace_fidelity.py, test_raw_output_hook_fidelity.py, test_responses_input_replay_ledger.py, test_glm_argument_notes.py, large_scale_serving/tests/test_gateway_request_identity.py): 1232 passed, 61 skipped, 0 failed in the 26.08 image.
• Real rollout, aggregated: GLM-5.2-NVFP4 TEP4, one GB300 node, two Codex CLI exec sessions + three probes (cap 16, no cap, chat max_tokens: 1). 31/31 checks: digests self-consistent 22/22 and count == usage.input_tokens 20/20; raw token_ids + finish_reason on 23/23 and len == usage.output_tokens; build_id on all 69 lines = provenance; caps echoed (16, default 4096), incomplete iff raw says length; thread prefix chaining 18/18, FIFO 18/18. Offline re-render (_create_input_tokens on the recorded bodies) reproduces the recorded prompt ids exactly: 20/20.
• Real rollout, disaggregated 1P1D (glm5.2_ctx_tep4 + glm5.2_gen_tep4, two nodes, the K3 fleet's image and TCP-only UCX list): attempt-1 exposed change (1) above (0/25 digests); attempt-2 on the fix: 31/31 checks, digests 22/22 from ctx_response, raw joined on disagg_request_id 23/23, re-render oracle 20/20.
• Two independent adversarial reviews of the diff before landing; every confirmed finding is fixed and has a test (details in the commit message and in linj's notes).

Verifier and smoke configs: /scratch/fsw/portfolios/coreai/users/linj/kernel_writer/tools/smoke/ (verify_traces.py is stdlib-only except the optional tokenizer/re-render part).

Not in this MR

• Kimi-K3 was not served in a smoke (GLM-5.2 only); its ledger paths are unit-tested.
• result.py: the in-proxy post-processing path (num_postprocess_workers: 0) never ends the raw hook on an error response, so a failed request's pending record lives until process exit (pre-existing; fleet configs run workers).
• Your patched NIXL UCX plugin is not readable by linj, so the disaggregated smoke ran with the stock plugin.

🤖 Generated with Claude Code

[JunyiXu-nv]

Requesting changes for two issues: _tail_frames now JSON-parses the whole stream (three times per response, on the event loop), and with capture on the raw records leave the directory analysis/raw_vs_parsed.py reads. Two questions and two nits inline. The commit also has no DCO Signed-off-by.

[linj-glitch]

Addressed the review in 622a8505e (and amended the base commit with the DCO Signed-off-by; history was force-pushed with lease). Unit tests in the 26.08 container on a GB300 node: 1,241 passed, 61 skipped, 0 failed (same file set as before plus test_raw_vs_parsed_layout.py). Summary of changes: _tail_frames no longer parses the stream body and the tail is split once per record; gateway_injected_terminal removed in favour of the access-log join; raw dumps readable from both layouts; GLM streaming path carries argument notes and normalization records dedupe per (call_id, event set); body_sha256 hashes the memoized request bytes; SPDX header added.