[iamwavecut]

Summary

Postgres ran on stock settings (128 MB shared buffers, 4 MB work_mem, HDD cost model), so most reads missed the buffer cache and sorts spilled to temporary files. Episode inserts also paid for a vector index that no query used.

Postgres settings (production compose)

• shared_buffers 128MB → 1536MB, effective_cache_size 6GB, work_mem 4MB → 16MB.
• random_page_cost 4 → 1.1 and effective_io_concurrency 1 → 200 (SSD-backed volume).
• jit=off: statements are short OLTP queries; JIT only added compile time on large estimates.
• wal_compression=lz4 (the image is built --with-lz4).
• auto_explain removed. It ran log_analyze with per-node timing on every statement, while the container's log driver is none, so its plans were never kept.
• shm_size: 1gb for parallel-query shared memory, stop_grace_period: 60s so a recreate shuts down cleanly instead of being killed after 10 s.
• synchronous_commit stays on.

Deploy applies Postgres configuration changes

deploy-production.sh started Postgres with --no-recreate, so new settings never reached a running database. It now compares docker compose config --hash postgresql with the running container's com.docker.compose.config-hash label. Only when they differ does it stop the app and recreate Postgres; an unchanged configuration leaves the database running.

Drop episode embeddings

Episodes are retrieved by full-text search only (SQL_RETRIEVE_MEMORY_EPISODES); no query reads memory_episodes.embedding. The memory pipeline no longer embeds episodes, and the episode insert has one code path without the vector.

• Migration 202 drops memory_episodes_embedding_hnsw_idx concurrently.
• Migration 203 drops the embedding column. Its TOAST space returns on the next table rewrite.
• Episode inserts no longer wait on the embedder, so an embedder outage no longer requeues a run at the episode step (cards still need it).

Migration compatibility

• 202 (-- no-transaction, one statement): DROP INDEX CONCURRENTLY IF EXISTS, so it does not block writes and is a no-op when rerun. Down recreates the HNSW index on the column 203's down restores.
• 203: DROP COLUMN IF EXISTS, a metadata-only change. Down re-adds embedding vector(512), the type migration 100 created. Data is not restorable: rollback gives an empty column, and the reverted code would embed new episodes again.
• Old binaries against the new schema: the previous release writes embedding on insert, so it must not run after 203 applies; the deploy stops the old app before the new one migrates.
• SQLx check: concurrent_index_migrations_are_single_statement_no_tx_files and sqlx_up_migration_versions_are_unique pass; the CI Postgres job applies all migrations (run_migrations_on) against pgvector/pg17 before the live storage tests.

Deployment risk and verification plan

• Risk: this deploy restarts Postgres once (config hash changes). The app is stopped first; expected DB downtime is the clean shutdown plus startup, seconds to a minute. Buffer cache starts cold.
• Risk: 1.5 GB shared buffers on a shared host with about 3 GB available memory; watch host memory after the restart.
• Migrations 202/203 are one-way for data: dropped episode embeddings are not restored by the down migrations.
• Plan after deploy: confirm the running image and that SHOW returns the new settings; confirm auto_explain is not loaded; confirm migrations 202/203 applied and the index is gone; compare buffer cache hit ratio, temp-file growth and episode insert time from pg_stat_statements before and after; check host memory and dialog failures for 30 minutes.

Verification

• cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace.
• docker compose config --hash postgresql on the production host matches the running container label for the current compose file, so an unchanged configuration is a no-op.
• bash -n deploy/production/deploy-production.sh.

[github-actions]

	Warnings
⚠️	Production deploy workflow/assets changed; include an explicit deployment-risk note and verification plan.
⚠️	Migration changes should explain forward/down compatibility, rollback/no-op behavior, or why rollback is impossible.
⚠️	Migration changes should mention a representative SQLx/storage/database check or explicitly justify why it is not needed.

	Messages
📖	Danger checked 8 changed file(s).

Generated by 🚫 dangerJS against 9aebeb5

[iamwavecut]

Danger warnings: the description now has a migration compatibility section (no-tx single-statement index drop, metadata-only column drop, down behaviour and the data that is not restorable, old-binary ordering) and names the SQLx checks; the deployment-risk note and post-deploy plan were already there.