Skip to content

feature/Runner-log-context · L-260916-b7759f — drop shim, json sink, no rich - #81

Merged
lchoquel merged 14 commits into
devfrom
feature/Runner-log-context
Sep 25, 2026
Merged

lchoquel merged 14 commits into
devfrom
feature/Runner-log-context

Conversation

@lchoquel

@lchoquel lchoquel commented Sep 19, 2026 •

Copy link
Copy Markdown
Member

The runner now binds the runtime's log context per request and emits its fields through the runtime's log calls, so its own request-id contextvars and logfmt flattener are gone. It boots on the json sink with the silent pretty-print mode and builds its image without the Rich extra; it pins pipelex 0.66.0, the release that carries the log seam.

Closes L-260916-b7759f

🤖 Generated with Claude Code

https://claude.ai/code/session_019vmb35cM2bCxTwn49kCAXQ


Summary by cubic

Replaces the runner's request-id contextvars and logfmt flattener with Pipelex's runtime log context and a JSON line sink on stderr. Error lines now carry each value as its own JSON field, and request_id is bound per request so every record the request emits — including ones Pipelex emits from inside a run — carries it. Drops the cli extra, so the Rich terminal renderer is never reached.

Closes L-260916-b7759f.

Notes

  • Anything parsing the old console/logfmt output must switch to the JSON line format.
  • A custom pipelex.toml must keep sink = "json", console_log_target = "stderr", and pretty_print_mode = "silent".
  • The sprint git-source pin for pipelex has collapsed onto the released 0.66.0; the runtime-contract guard and its test are deleted with it.
  • pipelex 0.66.0's redaction processor escapes control characters in field values and redacts credentials echoed into detail; tests and docs/logging.md cover both.
  • Local Docker builds no longer bake untracked .pipelex override files into the image.
  • Deferred review findings are tracked in wip/structured-logs/review-deferrals.md.

Written for commit ef292eb. Summary will update on new commits.

Review in cubic

lchoquel and others added 12 commits September 18, 2026 16:00
… boots on the json sink

Every value an error line carries is a record attribute now instead of a token in
a rendered `key=value` run: the handlers hand their fields to the runtime's log
call as `fields=`, `RequestIdMiddleware` binds `request_id` on the runtime's own
log context so it lands on every record emitted during a request — the ones
pipelex emits from inside a run included — and `.pipelex/pipelex.toml` selects
the `json` sink on stderr with a Rich-free pretty-print mode. The API's logfmt
flattener and its escaping go with the shim: a structured sink serializes a value
itself, so the message is now a short sentence built only from the status and the
error type, and no caller-supplied string reaches it.

`route` could not go where the item's text put it. `log.context` accepts exactly
three run identifiers — `request_id`, `pipeline_run_id`, `pipe_run_id` — and the
runtime is explicit that they are a closed reserved set spelled to match the
payload fields they come from. A route path is not one of them, so it travels as
a field on the call sites that want it, which is also the simplest thing: every
one of those sites already holds the `Request`.

`api/logging_context.py` is gone rather than rewritten. Its two contextvars also
fed `instance` and `request_id` into the problem documents the `api.errors`
helpers build, and those two members are now stamped by the handler that renders
the response, which is the frame that holds the `Request` — the same values, from
the same place, as the pipelex-error and request-validation paths already used.
The helpers stay parameter-clean and the API keeps no ambient request state.

The log seam is in no published pipelex release, so `pipelex` resolves from a git
source declared in `[tool.uv.sources]` for the duration of the sprint; the pin
site says what collapses it. Only uv reads that entry, so it steers the lock, the
sync and the Docker build and nothing else.

One adjacent fix, found while trying to run the image: `.dockerignore` did not
exclude `.pipelex/*_override.toml`, so a local `make docker-build` baked the
builder's own untracked overrides — storage backend, log level, telemetry
credentials — into the image, which then behaved differently from the one CI
builds out of a clean checkout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…his server is

Round 1 of the review confirmed seven findings, and what they have in common is
that the change described itself more confidently than it behaved.

The image context excluded one tier of per-developer configuration where the
runtime layers several: `_local`, `_{environment}` and `_temporary_override` too,
for every configuration family rather than for `pipelex` alone, plus two nested
under `inference/`. Environment names are open-ended, so the exclusion is now by
shape, with the one tracked variant named back in. Verified by building a
throwaway context and listing what survived.

The configuration test read the merged config while its comment claimed it pinned
the shipped file. The merged config is partly the developer's: a local
`sink = "console"` reds the suite while CI, which has no override file, stays
green. It reads the shipped file now, which is what the image boots on.

Three places said a boot selecting the console sink or the `rich` pretty-print
mode would refuse at startup. It would not: `typer` and `instructor` are core
pipelex dependencies that require Rich unconditionally, so Rich is in the image
and reachable, and these keys are the whole of what keeps it unused. The
Dockerfile already said this correctly and now the others agree with it.

`detail` was documented as a field every error line carries, but only the
failures this API authors itself attach it — a Pipelex `ErrorReport` has been
through disclosure redaction, so its body text is not the cause and is
deliberately not logged. Said so, in the page, the changelog and the repo guide.

The documented "replace the entire config directory" mount drops the shipped
`pipelex.toml` and silently returns the server to Pipelex's terminal-facing
defaults, which is precisely what this work exists to avoid. It now warns, and
names the keys to re-supply.

And the boot refuses a `pipelex` without `log.context`, by name. The branch the
pin names declares the same version as the published release, so no version
specifier can distinguish them and a `pip install .` resolves the wrong one;
without the check its first symptom is the error handler raising on the first
failure it tries to report.

Advances L-260916-b7759f

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sal that names the cause is reached

A published pipelex refuses this server's [runtime.log] sink key while api.main is being imported,
so the check in lifespan never ran on the install it exists for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019vmb35cM2bCxTwn49kCAXQ
…oes red

TestClient runs the app in a portal thread whose context never reaches the test's, so the release
test passed with a binding that was never released.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019vmb35cM2bCxTwn49kCAXQ
…text

# Conflicts:
#	pyproject.toml
#	uv.lock
`pipelex-api` takes `pipelex` from `pipelex`'s feature/Rich-behind-extra, written at [tool.uv.sources] of pyproject.toml with the lock regenerated in the same commit (P1). The collapse before this branch merges is `wt unpin _pipelex-api--runner-log-context pipelex --to <version>` (P2, P7).
`pipelex-api` takes `pipelex` from `pipelex`'s feature/Rich-behind-extra, written at [tool.uv.sources] of pyproject.toml with the lock regenerated in the same commit (P1). The collapse before this branch merges is `wt unpin _pipelex-api--runner-log-context pipelex --to <version>` (P2, P7).
…text

Absorbs dev's v0.26.1 through v0.27.5. The manifest keeps the branch's
sprint git source for `pipelex` over dev's `==0.65.0` pin, the lock keeps
that source and takes dev's package version, and the changelog keeps the
branch's Unreleased section above every released section dev added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TpAMq6PZNQfauyDvx3GR2H
`pipelex-api` takes `pipelex` from `pipelex`'s feature/Rich-behind-extra, written at [tool.uv.sources] of pyproject.toml with the lock regenerated in the same commit (P1). The collapse before this branch merges is `wt unpin _pipelex-api--runner-log-context pipelex --to <version>` (P2, P7).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TpAMq6PZNQfauyDvx3GR2H
@lchoquel

Copy link
Copy Markdown
Member Author

/rev round 4, profile 4, bar freeze on e4b607f (the branch after absorbing dev through v0.27.5 and moving the pipelex pin to 80896463).

Four reviewers ran: cubic, Codex review, Codex adversarial and code-review at level medium. code-review found nothing. Two findings were verified: Codex review's claim that .dockerignore drops a flavor's baked .pipelex/api_<env>.toml was refuted, because no tracked file is lost and the one real flavor builds from its own tree with its own ignore file. Codex adversarial's claim that a distributed /execute loses request_id at the worker was confirmed; it predates this branch and is not critical.

Nothing was fixed, because the bar admits only criticals and there were none. Every other finding is deferred with its trace in wip/structured-logs/review-deferrals.md under round 4 (fe2ac4b): the request-id gap on a Temporal /execute, the 413, 404 and 405 responses that write no api_error record, uvicorn's plain-text traceback on the JSON stderr stream, the missing version specifier on pipelex, a runtime-contract test that cannot fail, and a round-2 deferral whose premise dev has since closed.

converged — the round-4 pass at profile 4 was clean with full coverage, at or above the required depth of 4; nothing further is warranted

lchoquel and others added 2 commits September 25, 2026 18:43
`pipelex-api` takes `pipelex` from the registry again, with the lock regenerated in the same commit. The pin stood at 80896463 of `pipelex` (P7).
`wt unpin` replaced the git source and re-locked; this finishes the collapse. The requirement is the exact `==0.66.0` again, as this dependency is pinned everywhere else in the history, and the comments in `pyproject.toml` and the Dockerfile that explained the git source are gone with it. So is the runtime-contract guard, which refused a non-`uv` install resolving a published `pipelex` and was written to be deleted with the source, along with its test.

The release also carries the redaction processor, which escapes a control character in a field's value before any sink sees the record, so the crafted-detail test now expects the printable escape, a new test pins that a credential echoed into `detail` is redacted, and `docs/logging.md` documents both along with the trace-context keys. The changelog's temporary git-source entry becomes the pin entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TpAMq6PZNQfauyDvx3GR2H
@lchoquel
lchoquel merged commit dbb9edb into dev Sep 25, 2026
18 checks passed
@lchoquel
lchoquel deleted the feature/Runner-log-context branch September 25, 2026 16:51
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant