Skip to content

fix(logging): deliver quiet diagnostics through source builds (ENG-3579) - #12504

Merged
lucas-koontz merged 2 commits into
stagingfrom
fix/eng-3579-quiet-structured-logging
Oct 21, 2026
Merged

lucas-koontz merged 2 commits into
stagingfrom
fix/eng-3579-quiet-structured-logging

Conversation

@lucas-koontz

@lucas-koontz lucas-koontz commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

User story

As an operator building from source
I want quiet diagnostics in the assembled stack
So that I can identify failures without exposing request values

Why this matters

Operators do not receive upstream logging changes until the source gitlinks and images include them. The API and web images now build the coordinated candidate sources with WARNING/errors defaults. The build also passes source-derived package versions and uses the pinned server's canonical health and API routes.

Visual review

A deployed before/after capture remains a rollout gate. The real built-image smoke below verifies the candidate; it does not establish a production log-volume reduction.

Acceptance criteria

  • All four candidate gitlinks are recorded explicitly, including the data engine source.
  • The API wheel versions derive from the checked-out sources; missing or fallback versions fail the build.
  • Successful probes remain quiet, failed requests/readiness retain safe JSON, and nginx serves the API's JSON response rather than the UI fallback.
  • Build instructions, overrides/reset, isolated smoke testing and rollback are documented.
  • Upstream reviews/releases, final release pins, installed-stack behavior and collector/audit validation are complete.

How to test

  1. Initialize the pinned submodules and fetch their tags. Run COMPOSE_PROJECT_NAME=logging-review make docker-build; both images must build from source.
  2. Run python3 scripts/test-compose-logging.py --api-image logging-review-api --web-image logging-review-web. It uses disposable loopback-only containers without host data mounts.
  3. Confirm the reported agent/server versions match the pinned source. Verify readiness, a JSON response through nginx, a final refusal and a missing-index 503. The fixture request value must not appear in logs.
  4. Build the data engine separately from backend/data-vault/ with docker buildx build --load -t logging-review-data-vault -f docker/mindsdb.Dockerfile .. Check its baked health command, quiet successful /api/status and /api/util/ping, visible failed requests, request-ID echo and absence of private query values. This two-service Compose topology does not run it.
  5. Verify the installed deployment, production collector and retained operational evidence separately.

Notes for the reviewer

  • These are review candidate gitlinks. Replace them with the corresponding reviewed release commits before promotion. Advancing the older pins includes intervening upstream base changes, so the complete stack still needs release review.
  • Docker cannot see submodule Git tags in these source copies. Each wheel receives its own Hatch-derived version before dependency resolution. This prevents the old fallback Anton version from conflicting with the server's dependency floor.
  • Readiness and routing must follow the pinned server. /api/v1/health/ is the canonical endpoint; the previous /health probe failed, and /api/v1/ fell through to the UI.
  • Data-vault is source delivery only here. The gitlink advances, but this Compose topology runs API and web only.
  • Native startup warnings remain a collector boundary. Fresh SQLite initialization emitted a native SQLAlchemy reflection warning before the structured runtime events. The smoke does not claim every third-party stderr line is JSON.
  • Rollback restores the previous pins, images and settings together. No persistent host data or shared infrastructure was changed during validation.

Verified candidate sources

Component Reviewed source commit
Anton b73859e9fab97bfd430bf5829d012b9f96b473db
Server bececf886b489a218e69b7f40d76d7381451b462
Cowork 4bb38de0e4f16535ba3e2eac957d11ef406f7e76
Data engine 075c4e888b1daa15a9725662419cef17fa8e6d6a

Verified locally

These checks were repeated after updating all four gitlinks to the reviewed fixes. Images were built locally for arm64 and containers used loopback ports with no host data mounts; every smoke container was removed afterward.

Check Observed result
API/web image builds COMPOSE_PROJECT_NAME=eng3579-review-final-20261021 make docker-build: both passed.
API/web container smoke scripts/test-compose-logging.py passed the baked health command, API JSON through nginx, quiet probes, visible 401, web access statuses [200, 200, 503], private-value exclusion and graceful stop.
Installed API versions Anton 2.26.10.21.2rc3.dev2+gb73859e9f, server 0.26.10.21.2rc2.dev2+gbececf886, structlog 26.1.0.
Separate data-engine image Standard docker/mindsdb.Dockerfile build passed. Installed MindsDB 26.2.0, SQLAlchemy 2.0.55.
Data-engine container smoke Baked health command passed. /api/status and /api/util/ping returned 200 with no access records even under ACCESS_LOG_MODE=all; a missing API route retained a WARNING/404. Request IDs round-tripped and private query values were absent from logs.
Regression evidence The previous data-engine candidate emitted an INFO access record for its shipped /api/status health check; the final candidate no longer does.
Diff integrity git diff --check: passed.

The updated delivery commit 65ccb0a162d157a57c4e2a02701e208b1e463693 has a green hosted Snyk code check. The real builds and smoke evidence above are local. They do not establish provider-backed agent turns, installed-stack promotion, collector/export compatibility or a production log-volume reduction. Existing release pins outside this source build remain separate delivery gates.

Ships with

mindsdb/anton#621, mindsdb/cowork-server#741, mindsdb/cowork#1263, mindsdb/data-vault#5.

Merge/release order: mindsdb/anton#621, then mindsdb/cowork-server#741, then mindsdb/cowork#1263. mindsdb/data-vault#5 can proceed independently. Update reviewed release pins and rebuild #12504 last.
The source candidate pins include intervening upstream changes; reviewed release commits are required before promotion.

@lucas-koontz
lucas-koontz merged commit 5742050 into staging Oct 21, 2026
1 check passed
@lucas-koontz
lucas-koontz deleted the fix/eng-3579-quiet-structured-logging branch October 21, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants