[Evenss]

Summary

OpenFGA currently closes the shared SQLite connection pool before other services finish shutting down. This can cause Alertmanager snapshot writes or Tokenizer persistence to fail and may produce a nonzero exit code after SIGTERM.

This change keeps the shared pool owned by SigNoz, which closes it after all services stop. It also releases resources on initialization failure and makes repeated or concurrent shutdown calls idempotent while preserving error reporting.

Refs ob-labs/ai-vision-ob#110

Validation evidence

Validated on 2026-10-23 using Linux amd64 and Go 1.25.7. The recorded source files match the code committed in f80a8647f.

Regression tests

go test -mod=readonly -race \
  ./pkg/authz/openfgaserver \
  ./pkg/signoz \
  ./pkg/alertmanager/alertmanagerserver \
  ./pkg/tokenizer/opaquetokenizer \
  -count=1 -timeout=5m

Actual output:

ok  github.com/SigNoz/signoz/pkg/authz/openfgaserver            1.214s
ok  github.com/SigNoz/signoz/pkg/signoz                        1.225s
ok  github.com/SigNoz/signoz/pkg/alertmanager/alertmanagerserver 39.252s
ok  github.com/SigNoz/signoz/pkg/tokenizer/opaquetokenizer      1.202s

As a control, replacing only sqlstore.go with its version from base commit d560a6ad5 reproduced sql: database is closed in all three regression tests:

• TestSQLStoreClosePreservesSharedPool
• TestShutdownSnapshotsAfterOpenFGAStops
• TestShutdownFlushAfterOpenFGAStops

Evidence: tests/full-race.log, tests/baseline.log, and tests/summary.json in the evidence bundle.

Container shutdown and recovery

Built using the production Dockerfile:

docker buildx build --load --platform linux/amd64 \
  -f cmd/community/Dockerfile.ob \
  -t local/signoz-ob:issue110-acceptance .

Local image ID:

sha256:e35288c567fa0a6f9969d36c3663156f2fabe47150a572413b063dbde9c4f8d8

Round	Shutdown method	Duration	Exit code	OOMKilled
1	SIGTERM	0.990 s	0	false
2	docker stop	0.835 s	0	false
3	docker stop	0.744 s	0	false

Health, login, and authenticated API checks returned HTTP 200. Nonempty silence and notification-log fixtures survived restart, with snapshot timestamps advancing. No closed-database or snapshot-write errors were observed.

Evidence: runtime/summary.json, per-round container inspections, logs, and snapshot records.

Deployment and browser checks

• Compose: Frontend image replacement followed by another up completed successfully. Both SigNoz shutdowns exited 0; all six services were healthy afterward. Existing and newly ingested traces contained the expected three spans.
• CLI: Two same-image recast runs exited 0. All six previous containers exited 0 in each round. The historical trace retained its 20 expected spans.
• Persistence: Existing credentials, organization identity, SQL canary data, and storage bindings were preserved.
• Failure handling: A forced container exit of 137 caused the deployment wrapper to exit 1 and block replacement. Container IDs and mounts remained unchanged.
• Browser: The original account could log in and open the historical trace. Trace list, I/O, and detail requests returned HTTP 200, with no browser page errors.

Evidence: deployment/summary.json, deployment/negative-summary.json, cli/summary.json, browser-checks.json, and the browser screenshots.

Scope

Validation used locally built images and seekdb. Compose replaced the frontend image while restarting the same fixed SigNoz image; CLI validation also used the same images across recasts. Published images, cross-version SigNoz upgrades, and OceanBase CE remain unverified.

Snapshot checks establish fixture persistence and restoration; they do not cover concurrent snapshot updates or actual notification delivery.