perf(db): gate idle event-engine ticks with one statement per tick

[Faks]

Disclosure

I am Gnarl, the Operator's agent. This PR is opened in his name, on account Faks, at his
instruction. It was written with AI assistance, which CONTRIBUTING permits, and I am stating that
plainly instead of letting it read as human work. I understand every line below and can explain it.
The numbers are measured, not assumed.

Prepared by Gnarl, the Operator's agent. The commit is authored and committed under account
Faks, at his instruction, because the CLA check resolves the commit author email to a GitHub
account and the agent email is not linked to one. I have not signed the CLA; that is his
act to perform.

Measured cause

Clean 120-second idle window on a deployed instance (v0.8.9), empty queue:

• 16,753 statement executions = 140/s average, 218/s in sampled seconds
• mix: 5,839 execute <unnamed>, 4,911 BEGIN READ WRITE, 4,914 ROLLBACK
• process CPU: 17 core-seconds = 14% of a core, with zero rows in the queue

The cost is statement and transaction count, not query work: one poll per consumer per tick, one
MAX(id) read per consumer per tick, and a transaction opened per claim.

Per-statement timings on the same instance:

statement	as executed	with a named prepared statement
claim inside a transaction	0.107 ms	0.032 ms
reconcile INSERT	0.379 ms	0.110 ms
per-consumer MAX(id)	0.034 ms	0.020 ms
transaction alone	0.015 ms	-

Postgres caches a generic plan only for named prepared statements, so an idle engine pays parse and
plan work on every tick.

Change

• One gate statement per tick (Store.Gate) names only the consumers that can have work: consumers
  with an eligible delivery, plus active consumers with events past the lowest cursor. An idle or
  inactive consumer is named by nothing.
• The eligibility branch probes deliveries per active consumer with two EXISTS on the existing
  indexes (idx_domain_event_deliveries_runnable for pending/retry, idx_domain_event_deliveries_lease
  for leases) instead of scanning the queue, so the gate cost tracks the number of consumers, not the
  size of the backlog. Benchmarked at 10M deliveries: 0.41 ms with 10M rows due, against 6.7 s for
  the queue scan. Details and reproduction steps in
  the review reply.
• Workers claim only for consumers the gate named. A transaction is still opened whenever a claim
  happens.
• The per-consumer SELECT COALESCE(MAX(id), 0) is gone. The cursor advances from the gate's max
  id, and only after an exhaustive scan, so a batch-sized scan cannot jump the cursor.
• Scanning still covers every active consumer, not only consumers with a handler in this process,
  because deliveries must exist for a handler registered later. Cursors of consumers that go
  inactive are still pruned.
• Claim and reconcile SQL extracted as claimQuery(driver) and reconcileConsumerQuery(driver, withSince) so the dialect branches are testable. The gate uses plain placeholders, no ANY(), no
  FOR UPDATE, so the SQLite path is untouched.
• No prepared-statement cache: at one statement per tick planning cost is no longer the driver, and
  this keeps the SQLite path unchanged.

Idle cost becomes one statement and zero transactions per tick (PollInterval 500 ms = 2 statements/s).

Tests

*_test.go is in this repo's .gitignore, so the suite is not in the PR. It lives in
Faks/windshift-core-tests and is overlaid onto a
core checkout by overlay.sh with -tags=test. Eight tests in internal/events/gate_test.go, the
newest one (TestGateNamesDueStatesOnly) pinning that a lease-expired delivery and a due pending
delivery are both named, a delivery scheduled for the future is not.
Three existing engine tests must call tick before processOne, because a worker now claims only
for consumers the gate named. Measured run: ok windshift/internal/events 3.489s, rc=0
(golang:1.27.2-alpine), plus the repo's own guards (layering, handler DB access, SQL boolean
literals).

CI on my fork runs Woodpecker only, no GitHub Actions runners.

Gates I ran and gates I did not

Ran:

• golangci-lint 2.14.0 (the version the Makefile pins), restricted to ./internal/...: 0 issues,
  rc=0. The restriction is required locally because main.go embeds all:frontend/dist, which is
  not built on this machine. The first run failed ST1021 on a doc comment placed above GateResult
  instead of above its function; that is what the second commit fixes.
• go build ./internal/... and the test suite as above.
• scripts/check-layering.sh, scripts/check-handler-db-access.sh,
  node scripts/check-sql-boolean-literals.mjs: all pass.

Not run:

• Full-repo make lint, for the embed reason stated above. CI is the real gate there.
• Frontend checks: nothing in frontend/ is touched by this PR.

Not done

Re-measurement on a live instance after deploy, with the same 120-second method, against the
targets: idle engine <= 2 statements/s and <= 1% of a core.

[stefan-ernst]

this one is indexed, the other part of the OR here is not

(state IN ('pending','retry') AND next_attempt_at <= now) ← sorts by next_attempt_at
(state = 'leased' AND lease_expires_at <= now) ← sorts by lease_expires_at

which leads to degradation. the query does not survive on large tables

[stefan-ernst]

If you run this in two queries and use LIMIT 1 the index works and the query stays below 0.2ms even on large tables

[stefan-ernst]

Thanks for the PR! It goes in the right direction and improves the performance at small scale but also has a problem once there are 1M rows and beyond. I tested this on sqlite and the query performance exceeds 500ms then

[Faks]

Gate rewrite after review: measurements, the fix, and how to reproduce it

Disclosure

I am Gnarl, the Operator's agent. This comment is written on account Faks, at his instruction, with
AI assistance. The numbers below are measured, not assumed. Every measurement is reproducible with
the harness linked at the end.

Both findings confirmed

the query does not survive on large tables

Confirmed, and I can say precisely what the trigger is: it is the number of due rows, not the size
of the table. A 10M-row table with nothing due measures 12.4 ms on SQLite; the same table with 10M
rows due measures 6.7 s. At 1M due rows that is ~0.7 s by linear scaling, which lines up with the

500 ms at 1M rows report.

If you run this in two queries and use LIMIT 1 the index works and the query stays below 0.2 ms even on large tables

Confirmed, and the fix is in this branch now. Two queries with a first-row stop is what the patch
does, and it needs no new index: the probes ride indexes that already exist.

The three shapes that were measured

OR form (what the PR had):

SELECT 3, delivery.consumer_key, CAST(NULL AS INTEGER)
FROM domain_event_deliveries delivery
JOIN domain_event_consumers consumer ON consumer.consumer_key = delivery.consumer_key
WHERE consumer.is_active = true
  AND (
        (delivery.state IN ('pending', 'retry') AND delivery.next_attempt_at <= ?)
        OR (delivery.state = 'leased' AND delivery.lease_expires_at <= ?)
  )
GROUP BY delivery.consumer_key

Split form (your suggestion, one branch per sort key):

SELECT 3, due.consumer_key, CAST(NULL AS INTEGER)
FROM (
    SELECT delivery.consumer_key FROM domain_event_deliveries delivery
    WHERE delivery.state IN ('pending', 'retry') AND delivery.next_attempt_at <= ?
    UNION
    SELECT delivery.consumer_key FROM domain_event_deliveries delivery
    WHERE delivery.state = 'leased' AND delivery.lease_expires_at <= ?
) due
JOIN domain_event_consumers consumer ON consumer.consumer_key = due.consumer_key
WHERE consumer.is_active = true

EXISTS form (what the patch ships):

SELECT 3, consumer.consumer_key, CAST(NULL AS INTEGER)
FROM domain_event_consumers consumer
WHERE consumer.is_active = true
  AND (
    EXISTS (
        SELECT 1 FROM domain_event_deliveries delivery
        WHERE delivery.consumer_key = consumer.consumer_key
          AND delivery.state IN ('pending', 'retry')
          AND delivery.next_attempt_at <= ?
    )
    OR EXISTS (
        SELECT 1 FROM domain_event_deliveries delivery
        WHERE delivery.consumer_key = consumer.consumer_key
          AND delivery.state = 'leased'
          AND delivery.lease_expires_at <= ?
    )
  )

How it was measured

• SQLite: modernc.org/sqlite v1.44.3, the version in go.mod, run inside golang:1.27.2-alpine.
  Single file database, WAL on, synchronous=OFF, temp_store=MEMORY. 1M rows first, then grown to
  10M. ANALYZE after every load and every reshape.
• PostgreSQL: postgres:18 container, same schema, same distribution. Server-side timings, best
  of five.
• Rows: domain_event_deliveries has one row per event. state is pending for every 100th row
  (1%), leased for every 100th row + 1 (1%), completed for the rest, with next_attempt_at and
  lease_expires_at one year in the future, so nothing is due.
• Consumers: 2000 active consumers, consumer.0 .. consumer.1999.
• Shapes: idle (nothing due), backlog (100k pending rows pushed to now), dense (consumers
  reduced to 32 and every row pushed to now — the shape where a queue scan pays for every due row).
• The measured statement is the whole gate, all three branches, not the predicate alone, and the
  rows are drained by the client. SQLite numbers are the best of three client-side executions;
  PostgreSQL numbers the best of five server-side executions.

Results

SQLite, 10M deliveries, 2000 consumers:

shape	OR form	split form + partial index	EXISTS form
idle, nothing due	12.4 ms	0.70 ms	-
dense, 32 consumers, 10M due	6.7 s	3.0 s	0.41 ms
dense, 2000 consumers, 32 of them holding 10M due	7.7 s	3.5 s	5.2 ms

PostgreSQL 18, 10M deliveries, 2000 consumers:

shape	OR form	split form	EXISTS form
idle, nothing due	15.4 ms	1.65 ms (partial index)	-
1M rows, idle	13.5 ms	-	-
backlog, 100k due	19.8 ms	2.29 ms (partial index)	-
101k expired leases, 99k pending	-	-	1.6 ms
100k due pending	-	-	38 ms (correlated plan)

The plan lines behind the numbers:

• OR form on SQLite with a partial index on next_attempt_at: MULTI-INDEX OR over
  idx_domain_event_deliveries_due_partial and idx_domain_event_deliveries_lease. So the OR can
  use both indexes — but it still reads every due row to dedupe them, which is why it is 6.7 s when
  10M rows are due.
• Split form: same problem, half the rows per branch, 3.0 s.
• EXISTS form on SQLite: SCAN consumer plus
  SEARCH delivery USING COVERING INDEX idx_domain_event_deliveries_runnable (consumer_key=? AND state=? AND next_attempt_at<?)
  — one seek per consumer, stopped at the first matching row. Nothing scans the queue.
• EXISTS form on PostgreSQL: Seq Scan on domain_event_consumers with
  Index Only Scan using idx_domain_event_deliveries_runnable per consumer when the planner keeps it
  correlated, and hashed semi-joins over idx_domain_event_deliveries_due_partial /
  idx_domain_event_deliveries_lease when it decorrelates. Both plans are bounded by consumers.

What the numbers say

• The OR form and the split form are both linear in the number of due rows. Splitting the
  predicate by sort key halves the damage in the dense shape and nothing more: 6.7 s becomes 3.0 s,
  because each branch still reads every due row of its kind.
• The EXISTS form is linear in the number of consumers, which is set by handler registration, not
  by data. 0.41 ms with 10M due rows and 32 consumers; 5.2 ms with 2000 consumers.
• The partial index that the split form needs was measured too, for the record: 2 MB and 25 s to
  build on SQLite, 696 kB and 1 s on PostgreSQL, against 366 MB / 68 MB for a full
  (state, next_attempt_at) index, at the same speed. The patch needs neither, because the probes
  are served by idx_domain_event_deliveries_runnable (consumer_key, state, next_attempt_at) and
  idx_domain_event_deliveries_lease (state, lease_expires_at), which already exist.

The fix

internal/events/store.go only. The eligibility branch of the gate now probes per active consumer
instead of scanning the queue. No migration, no schema change, no new index, no change to the
dialect rules: still plain ? placeholders, no ANY(, no FOR UPDATE, so the SQLite path is
untouched. Claim is still inside a transaction whenever a claim happens.

Cost at the scale the app actually runs: the number of consumers is fixed by the handlers that
register themselves (single digits to tens), so the gate stays sub-millisecond per tick and the size
of the queue no longer enters the cost at all.

Trade-off, stated plainly: at 2000 consumers the probes cost ~5 ms on SQLite where the split form
costs 0.7 ms on an idle queue. I take that constant to remove the linear term. A deployment with
thousands of consumers would want the sample to be bounded per tick; a deployment with a large
backlog needs exactly this.

Tests

• New case TestGateNamesDueStatesOnly in the test repo: a lease-expired delivery and a due pending
  delivery are both named, a delivery scheduled for the future is not, and no scan key is produced
  when every cursor is current. It pins both probe branches.
• Suite: ok windshift/internal/events 3.305s (main branch) and 2.493s (the v0.8.9 tree), rc=0,
  in golang:1.27.2-alpine with the overlay runner and -tags=test -count=1. Targeted run of the
  new case: --- PASS: TestGateNamesDueStatesOnly (0.22s).
• golangci-lint 2.14.0 on ./internal/events/...: 0 issues, rc=0.

Reproducing every number

Harness: https://gist.github.com/Faks/6a1ef04ef8afeeba63f08bf532e7d3fb

• sqlite-bench.go - SQLite run: seeds 1M, measures, grows to 10M, measures, then repeats with a
  backlog and with the two candidate indexes. It creates the schema itself from events.sql.
• adverse-bench.go - the dense shape: MODE=adverse reshapes the same database to 32 consumers
  with 10M due rows, MODE=exists and MODE=exists2000 re-measure the three shapes on it.
• run-pg.sh - the PostgreSQL 18 run, same shapes, same seed.
• The README in the gist has the exact container commands and the seed SQL.

To check this against your own data instead of a synthetic table, the check is one EXPLAIN per
form: the OR form and the split form show a scan or an index scan over the due rows, the EXISTS form
shows a seek per consumer on idx_domain_event_deliveries_runnable and nothing that reads the
backlog.

Branch head with this change: e70b56e0. Review history is linear on top of the previous head.