aqp-engine

An Approximate Query Processing (AQP) engine written in Rust. It sits in front of PostgreSQL for the parts of an application that aggregate (dashboards, reports, counters), speaks PostgreSQL's own network protocol, and answers their queries in milliseconds from an in-memory random sample, with an honest error margin. The sample follows every INSERT, UPDATE and DELETE live.

Exact queries (SELECT * ..., INSERT, JOIN...) pass straight through to PostgreSQL. Aggregate queries (AVG, SUM, COUNT, COUNT(DISTINCT), with WHERE and GROUP BY) are estimated from the sample with the Central Limit Theorem and HyperLogLog, and every answer says how far off it can be.

$ psql "host=127.0.0.1 port=6432 user=aqp dbname=aqp" -c "SELECT region, COUNT(*) AS orders, AVG(amount) FROM sales GROUP BY region"
NOTICE:  aqp-engine: approximate answer (95% confidence, margin of error up to ±3.3%), from 100,000 of 100,000 sampled rows of a 5,000,000-row table; live, synced 0.5 s ago
 region | orders  |    AVG(amount)
--------+---------+--------------------
 APAC   | 1010100 |  400.1514612414599
 EU     | 1487100 | 408.59997511935796
 LATAM  |  497450 |  397.6164870841281
 MEA    |  246150 |  399.9161019703439
 NA     | 1759200 |  406.3432028763096
(5 rows)

Status: working prototype with the core pieces a production deployment needs (live change tracking, crash-safe snapshots, the PostgreSQL protocol, TLS, SCRAM authentication), verified end to end in a local lab. It has not been battle-tested under real production load; see What is supported, and what is not.

---

The idea in plain words

To find out what share of a country votes for a candidate, nobody asks all 15 million voters. Pollsters ask about 1,000 people chosen at random and publish something like "42% ± 3%": fast, cheap, close enough, and honest about how close.

aqp-engine does the same with a table. Instead of reading 5 million rows to compute an average, it reads a random sample of 100,000 rows that it keeps in memory, and answers "133.06 ± 0.64" in about a millisecond.

Each part of the engine exists because this idea raises a question.

Which questions can a poll answer? "What is the average age?" can be answered from a poll; "what is John's phone number?" cannot, and approximating it would be a serious mistake. So the intent detector and the planner decide which queries the sample is allowed to answer.

What happens to the rest? They still deserve an answer, and the right one. The router sends them to PostgreSQL unchanged.

Who is in the poll? A biased sample gives biased answers, so the sampler picks rows fairly: every row of the table has the same chance of being chosen.

How wrong can the answer be? A number without a margin is just a guess. The math turns the sample into an estimate and says how far off it can be: the Central Limit Theorem for averages and totals, HyperLogLog for distinct counts.

What if people change their minds after the poll? The answers would slowly go stale, so the sample follows every INSERT, UPDATE and DELETE as it happens.

Does the poll have to be redone after every restart? On a big table that would mean minutes of waiting, so the sample is saved to disk and reloaded in a fraction of a second.

How do other programs ask their questions? A prompt only works for a person, so the engine also speaks PostgreSQL's network protocol, and any PostgreSQL client can connect to it.

	Exact (PostgreSQL)	Approximate (aqp-engine)
Rows read per query	All of them (N)	The sample (n), already in memory
Latency	Grows with the table	Depends on n only
Result	Exact	Estimate ± margin of error
Good for	Invoices, balances, anything legal or financial	Dashboards, exploration, trends, "roughly how many?"

---

Two knobs: how sure, and how precise

Every answer comes with a margin, like the poll's "± 3%". Two settings shape it, and they are easy to mix up because both change the number after the ±:

margin = z × s / √n      z = how sure we want to be            AQP_CONFIDENCE
                         n = how many rows the sample holds     AQP_SAMPLE_SIZE
                         s = how spread out the data is         (nobody chooses this one)

Knob 1: the confidence level (AQP_CONFIDENCE)

It says how often the true value must fall inside the margin. It does not bring the estimate any closer to the truth: it draws a wider range around the same estimate so that the range misses less often. Think of a friend telling you when they will arrive. "Between 3 and 4" turns out right most days; "between 2:45 and 4:15" is right almost every day. Same friend, same traffic, a wider promise. That is why this knob is free: same sample, same query time, same memory.

AQP_CONFIDENCE	z	Margin compared with 95%	Measured: the margin contained the truth
0.90	1.645	0.84×	90.5%
0.95 (default)	1.960	1×	95.4%
0.98	2.326	1.19×	97.4%
0.99	2.576	1.31×	98.9%
0.999	3.291	1.68×	99.5%

The last column comes from 1,000 samples of 400 values drawn from a long-tailed population (the check the tests run). At 99.9% the margin falls a little short: that far out in the tails, the average of 400 skewed values is not quite bell-shaped yet.

Knob 2: the sample size (AQP_SAMPLE_SIZE), the one that matters most

This is the knob that buys precision. The margin shrinks with √n: four times more rows halve it. The price is paid in query time and memory, which grow at least as fast as n. So the curve drops steeply at first and then flattens out, while the costs keep climbing:

Measured on the 5-million-row lab table with SELECT AVG(price) FROM sales WHERE region = 'EU' (every number is in Benchmark results):

• From 25,000 to 100,000 rows (×4), the margin halves, from ±1.77% to ±0.87%, and memory goes from 11 to 39 MB. Queries stay under 2 ms. Cheap precision.
• From 100,000 to 800,000 rows (×8), the margin only drops to a third (±0.29%), while memory grows 7 times (285 MB) and the query time 17 times (25 ms). Expensive precision.

Why this is the knob that matters most:

1. It is the only one that changes the error itself. The typical distance between the estimate and the truth is s/√n, and z is not in it. Raising the confidence moves the promise; enlarging the sample moves the estimate.
2. The margin depends on the rows that match, not on the whole sample. WHERE region = 'MEA' keeps 5% of the table, so it is answered from about 5% of the sample: 5,000 of 100,000 rows. A GROUP BY splits the sample among its groups. Below 30 matching rows the engine does not estimate at all and sends the query to PostgreSQL. So the sample size decides how narrow a question can still be answered quickly: with 100,000 rows, a filter has to keep at least about 0.03% of the table (30 rows); with 25,000, at least 0.12%.
3. Confidence ends up costing sample size anyway. Going from 95% to 99% widens the margin 1.31 times; to keep the old margin you need 1.31² ≈ 1.7 times more rows.

Where the costs land. They are not paid in the same place, nor by the same people:

Cost	Grows with n?	Who pays it, and when
Query time	Yes: 0.85 ms at 25,000 rows, 1.5 ms at 100,000, 25 ms at 800,000	Every approximate query, whoever is waiting for it (a person looking at a dashboard). Every sampled row is read once per query. Past 200,000 rows it grows a bit faster than n, probably because the sample no longer fits in the processor's caches. Even 25 ms is 8 to 18 times faster than PostgreSQL's 213–446 ms on the same table.
RAM	Yes: about 300 bytes of data per sampled row (7 sampled columns here); the whole process holds about 1.25× that, plus a few MB of its own	The machine running the engine, all the time, once per table in AQP_TABLES. Wider tables take more per row.
Snapshot file	Yes: about 70 bytes per row (1.7 MB at 25,000 rows, 51 MB at 800,000)	The disk, and the time to save it: it is rewritten every AQP_SNAPSHOT_INTERVAL_SECS (30 s) when the sample has changed, and read on every restart (0.2 s at 100,000 rows).
First scan	No: 13–16 s at every size	Once, the very first time: the table is read whole whatever n is, because every row needs its chance to be picked.
Each live change	No	An INSERT, UPDATE or DELETE touches at most one sampled row.

How to choose. Start with the default, 100,000 rows: about ±0.9% on a filtered average, 1.5 ms, 39 MB. Raise it when the questions are narrow (selective filters, many or rare groups) and margins come out wide or queries fall back to PostgreSQL. Lower it when memory is tight: 25,000 rows still give about ±1.8% in under a millisecond. Raise the confidence only when a miss is expensive; it costs no memory, only a wider margin.

---

Why a proxy is worth an extra hop

A proxy adds one network hop, and every query that goes through it pays for it. Measured with a real PostgreSQL client over TCP, on one machine (details):

	Straight to PostgreSQL	Through aqp-engine
An aggregate over 5 million rows (approximated)	262 ms	2 ms (128× faster)
A dashboard with 10 such charts (an illustration)	10 × 262 ms ≈ 2.6 s	10 × 2 ms ≈ 20 ms
A short query it does not approximate, a lookup by id (passed through)	0.43 ms	0.57 ms (+0.14 ms)
A large exact result, 100,000 rows (passed through)	86 ms	155 ms (+69 ms)

It is worth it for three reasons:

1. The gain and the cost are of different sizes. The hop costs a fixed fraction of a millisecond; an approximated query saves a whole table scan, and that saving grows with the table. One approximated query pays for about 1,800 pass-through hops.
2. It takes load off PostgreSQL too. An approximated query never reaches PostgreSQL, so the scan it would have done (hundreds of milliseconds of CPU and disk) is left free for the queries that need exact answers. The engine's only regular load on PostgreSQL is reading the change stream once a second.
3. You decide where the hop happens. Only the features that aggregate use the engine's connection string (next section), so a login or a checkout never pays it. The one real cost left is large pass-through results, which the engine collects before sending: keep them on the direct connection (streaming them is on the roadmap).

The hop can also be made smaller: run the engine on the same machine as the application, or call the Rust library in-process (no hop; deciding the route takes 5.8 µs).

---

Where it fits: only the parts of an application that aggregate

In plain words. aqp-engine is not meant to sit in front of a whole application. It is meant for the parts that count, sum or average many rows: dashboards, reports, statistics. Everything else (logging in, saving an order, reading one document) keeps talking to PostgreSQL directly, exactly as before, and pays nothing.

"But how do I know in advance which queries are analytical?" Queries do not appear at random: every query is written by a programmer, for one specific feature. Whoever writes the login knows it looks up one user; whoever writes the admin dashboard knows it counts things. So the choice is made once, per feature, when the code is written, not query by query at run time:

Feature	What its queries do	Connection it uses
Log in, sign up	read one user by email	DATABASE_URL → PostgreSQL
Upload a document, place an order	insert and update specific rows	DATABASE_URL → PostgreSQL
Payments, subscriptions	update one account	DATABASE_URL → PostgreSQL
Admin dashboard, reports, visitor counters	count and sum millions of rows	ANALYTICS_URL → aqp-engine

In code, that is two connections, and each function uses the one that fits it:

transactional_db = connect(DATABASE_URL)   # straight to PostgreSQL
analytics_db = connect(ANALYTICS_URL)      # through aqp-engine

def login(email):
    # One user, found by its email: PostgreSQL answers it in under a millisecond,
    # and this function never goes through aqp-engine.
    return transactional_db.query("SELECT * FROM users WHERE email = %s", email)

def admin_dashboard():
    # Counts over millions of rows: aqp-engine answers from its sample in a few
    # milliseconds, with the margin of error, instead of a full scan.
    return analytics_db.query("SELECT region, COUNT(*) FROM sales GROUP BY region")

(connect and query stand for your database library. With psycopg2 they are psycopg2.connect(url) and cursor.execute(sql, params); ANALYTICS_URL is for example postgresql://aqp:secret@127.0.0.1:6432/aqp.)

flowchart LR
    subgraph App["Your application"]
        Tx["login · uploads · payments<br/>(transactional)"]
        An["dashboards · reports · counters<br/>(analytical)"]
    end
    Tx -- "DATABASE_URL" --> PG[("🐘 PostgreSQL")]
    An -- "ANALYTICS_URL" --> AQP["🦀 aqp-engine"]
    AQP -- "approximates what it can" --> Fast(["answer in ~2 ms"])
    AQP -- "forwards the rest (+0.14 ms)" --> PG

    classDef exact fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef approx fill:#dcfce7,stroke:#16a34a,color:#14532d
    class Tx,PG exact
    class An,AQP,Fast approx

Loading

With this split, the hop is paid only where it saves hundreds of milliseconds, and the rare query in an analytical feature that the engine cannot approximate costs +0.14 ms.

Two levels of decision. The split above is decided by you, once, per feature. Inside the analytical part, the engine then decides by itself, per query (in 5.8 µs) whether it can approximate it; if not, it forwards the query to PostgreSQL, so an analytical feature never breaks because of a query the engine does not support.

When you really cannot know in advance, as with a BI tool (Metabase, Grafana) or an analyst typing SQL: point the whole tool at the engine. Every aggregate of a supported shape is answered from the sample, and everything else goes to PostgreSQL for +0.14 ms, which a person looking at a screen does not notice.

What to keep away from it: the high-volume transactional path, such as the login that runs thousands of times a minute. If it reaches the engine by mistake, nothing breaks, it only costs +0.14 ms per query (and large results are collected before being sent). An analytics-only mode that refuses such queries, to make the mistake visible, is on the roadmap.

---

Benchmark results

In short, measured on a 5-million-row table (the setup and every query are below):

	PostgreSQL (exact)	aqp-engine (approximate)
AVG, SUM, COUNT, with and without WHERE	213–446 ms	1–4 ms (63× to 431× faster)
GROUP BY region	523 ms	3.4 ms (154× faster)
COUNT(DISTINCT customer_id)	1,847 ms	0.48 ms (3,839× faster)
Error of the estimates	0	0.14% to 2.28%
Estimates inside their own ±95% margin	–	12 of 13 (about 19 of 20 is what 95% means)
Start-up	–	0.2 s from the saved sample (one 12–18 s scan the very first time)
After INSERT of 100,000 rows	–	reflected in the estimates within 2.5 s
Cost of the proxy on a short pass-through query	–	+0.14 ms over the network (details)
Sample size from 25,000 to 800,000 rows	–	margin ±1.77% → ±0.29%, query 0.85 → 25 ms, RAM 11 → 285 MB (details)

Setup: Apple M1 (8 cores); PostgreSQL 17 in Docker with 2 CPUs and 2 GB of RAM for the Docker VM, default settings (parallel query on); 5,000,000 rows (508 MB); sample of 100,000 rows (2%), seed 42, loaded from the saved snapshot in 0.2 s; median of 3 runs; release build.

Query	Exact (PostgreSQL)	Approximate (aqp-engine)	Real error	Inside the ± margin?	PostgreSQL	aqp-engine	Speedup
SELECT AVG(price) FROM sales	132.73	133.06 ± 0.64	0.25%	yes	445.99 ms	1.04 ms	431×
SELECT SUM(amount) FROM sales	2,020,157,553.78	2,022,894,646.50 ± 14,781,844.54	0.14%	yes	213.01 ms	1.19 ms	179×
SELECT COUNT(*) FROM sales WHERE region = 'EU'	1,500,598	1,487,100 ± 14,024	0.90%	yes	257.14 ms	2.08 ms	123×
SELECT AVG(amount) FROM sales WHERE category = 'electronics'	404.36	405.32 ± 5.41	0.24%	yes	261.17 ms	1.66 ms	157×
SELECT SUM(quantity) FROM sales WHERE price > 100	7,336,037	7,354,250 ± 65,904	0.25%	yes	286.70 ms	2.21 ms	130×
SELECT AVG(price) FROM sales WHERE region IN ('LATAM', 'MEA') AND quantity >= 3	132.49	131.53 ± 2.39	0.72%	yes	227.57 ms	3.29 ms	69×
SELECT COUNT(*) FROM sales WHERE price BETWEEN 50 AND 60	334,474	326,850 ± 7,583	2.28%	no	255.05 ms	4.08 ms	63×
SELECT COUNT(DISTINCT customer_id) FROM sales	200,000	201,519 ± 3,209	0.76%	yes	1,846.71 ms	0.48 ms	3,839×
SELECT region, AVG(amount) FROM sales GROUP BY region	5 groups	5 groups	up to 1.37%	5 of 5	523.34 ms	3.39 ms	154×

How to read it:

• One estimate missed its margin, and that is expected. A 95% interval misses about 1 time in 20; here 1 of 13 estimates did, by 41 rows (error 7,624, margin 7,583). The tests check the 95% claim directly: over 1,000 samples of a skewed population, the margin contained the true mean 92–98% of the time; with z = 1.0 instead of 1.96 it dropped to 68.6%, as the normal distribution predicts.
• The speedup grows with the table: PostgreSQL's scan grows with N, the engine's time depends on n only.
• COUNT(DISTINCT) is the biggest win: PostgreSQL sorts or hashes 5 million values; the engine reads 16 KB.
• The engine's first query after start-up is slower (10–20 ms) while CPU caches warm up.

Through the network: what the proxy costs

The table above times the engine inside its own process. A proxy also adds a network hop, so the benchmark measures it too: a real PostgreSQL client over TCP (on the same machine), once straight to PostgreSQL and once through the engine's server. Median of 30 runs (5 for the large result):

Query	Straight to PostgreSQL	Through aqp-engine	Difference
SELECT AVG(price) FROM sales (approximated)	262.02 ms	2.04 ms	128× faster
SELECT id, region, price FROM sales WHERE id = 42 (passed through)	0.43 ms	0.57 ms	+0.14 ms (+33%)
SELECT 1 (passed through)	0.39 ms	0.38 ms	no measurable difference
SELECT * FROM sales WHERE id <= 100000: first row	0.58 ms	135.38 ms	+134.80 ms
SELECT * FROM sales WHERE id <= 100000: last row	86.13 ms	155.20 ms	+69.07 ms (+80%)

• Approximate answers stay far ahead: the network adds about 1 ms to the engine's 1 ms.
• The hop itself is cheap: +0.14 ms on a short lookup, nothing measurable on SELECT 1. Deciding the route takes 5.8 µs. Across machines in a data centre, add the network round trip (typically 0.2–1 ms).
• Large pass-through results are the real cost today: the engine collects the whole result before sending it, so the first of 100,000 rows arrives after 135 ms instead of 0.6 ms. Streaming rows as they arrive is the first item of the roadmap.
• Where to put it so it only costs where it helps: see Running it in production.

Sample size: what each extra row costs and buys

The same query at six sample sizes: SELECT AVG(price) FROM sales WHERE region = 'EU' (about 30% of the rows match), 95% confidence, seed 42, median of 50 runs, release build. Each size is measured by two fresh processes: one builds the sample and times the query; the other starts the way the engine normally does, from the saved snapshot, and reads the memory of the whole process (with macOS's footprint, the figure Activity Monitor shows; ps leaves out compressed memory and reads too low on a Mac).

Sample size	Margin (±%)	Real error	Query time	Sample data in memory	Engine process (RAM)	Snapshot file
25,000	1.77%	0.03%	0.85 ms	7.3 MB	11 MB	1.7 MB
50,000	1.23%	0.10%	0.73 ms	14.5 MB	20 MB	3.3 MB
100,000 (default)	0.87%	0.90%	1.50 ms	28.8 MB	39 MB	6.5 MB
200,000	0.61%	0.26%	4.63 ms	57.5 MB	73 MB	12.9 MB
400,000	0.42%	0.02%	11.03 ms	115.0 MB	144 MB	25.6 MB
800,000	0.29%	0.04%	25.16 ms	229.8 MB	285 MB	51.1 MB

• The margin follows √n: every ×4 in rows halves it (1.77% → 0.87% → 0.42%).
• Memory and the snapshot follow n: they double every time the sample doubles.
• The query time stays under 2 ms up to 100,000 rows, then grows 2.3 to 3.1 times per doubling.
• The real error is one draw per size, so it jumps around. At 100,000 rows the estimate missed its margin by a hair (0.90% against ±0.87%), the kind of miss 95% allows about 1 time in 20; every other size landed well inside.
• The first scan took 13–16 s at every size: it reads the whole table whatever n is.

The chart in Two knobs is drawn from these numbers (docs/sample-size.csv).

---

Architecture

flowchart TB
    Clients(["👤 psql · BI tools · drivers"]) -- "PostgreSQL protocol<br/>SCRAM password" --> Server
    Prompt(["⌨️ aqp&gt; prompt"]) --> Router

    subgraph Engine["🦀 aqp-engine"]
        Server["Server<br/>one session per client"] --> Router
        Router{"Intent + planner<br/>sample or PostgreSQL?"}
        Router -- "approximate" --> Math["Math engine<br/>CLT · HyperLogLog · GROUP BY"]
        Samples[("Samples in memory<br/>by column + sketches")] --> Math
        Sync["Sync task<br/>applies changes<br/>(random pairing)"] --> Samples
        Sync <--> Disk[("💾 Snapshot file")]
    end

    Router -- "exact: SQL unchanged" --> PG[("🐘 PostgreSQL")]
    PG -- "change stream<br/>(replication slot)" --> Sync
    PG -. "one full scan,<br/>first start only" .-> Samples

    classDef exact fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef approx fill:#dcfce7,stroke:#16a34a,color:#14532d
    classDef neutral fill:#f3f4f6,stroke:#6b7280,color:#111827
    class PG exact
    class Math,Samples,Sync,Disk approx
    class Server,Router neutral

Loading

• Blue: PostgreSQL, the source of truth. Exact queries go there unchanged.
• Green: the approximate path. At query time PostgreSQL is not contacted; in the background the sync task keeps the samples in step with the table.
• Whenever the approximate path cannot give a trustworthy answer, the query goes to PostgreSQL, and the reason is reported.

---

How a query is answered

1 · Parser, AST and intent

In plain words. A query arrives as text. The engine turns it into a tree where every node has a meaning (a function, a column, a comparison), and looks for AVG, SUM or COUNT in it.

Technical details. sqlparser-rs (PostgreSQL dialect) builds an AST (Abstract Syntax Tree) of Rust enums and structs. For SELECT AVG(price) FROM sales WHERE region = 'EU':

graph TD
    Q["Statement::Query"] --> S["SetExpr::Select"]
    S --> PR["projection"]
    S --> FR["from"]
    S --> WH["selection (WHERE)"]
    PR --> FN["Expr::Function<br/><b>AVG</b>"]
    FN --> ARG["Expr::Identifier<br/>price"]
    FR --> T["TableFactor::Table<br/>sales"]
    WH --> BO["Expr::BinaryOp<br/>="]
    BO --> L["Expr::Identifier<br/>region"]
    BO --> R["Expr::Value<br/>'EU'"]

    classDef hit fill:#dcfce7,stroke:#16a34a,color:#14532d,stroke-width:2px
    class FN hit

Loading

detect_intent walks the SELECT list with match (pattern matching) and searches recursively, so ROUND(AVG(price), 2) is recognised as an aggregate query too. Window functions (COUNT(*) OVER ()) are not aggregates here: they return one value per row. \ast <sql> prints the tree for any query.

Code: src/intent.rs.

2 · Planner and router

In plain words. Wanting an aggregate is not enough: the sample must be able to answer this exact query. If it cannot, the SQL goes to PostgreSQL untouched, with the reason.

Technical details. The planner accepts exactly these shapes:

SELECT <aggregate> [, <aggregate> ...] FROM <one table> [WHERE <condition>]
SELECT <column>, <aggregate> [, ...]   FROM <one table> [WHERE <condition>] GROUP BY <column>

Everything else is sent to PostgreSQL with a readable reason: JOIN, HAVING, sub-queries, ORDER BY, LIMIT, DISTINCT, WITH, SELECT INTO, TABLESAMPLE, row locks, GROUP BY of several columns, and aggregates inside other expressions such as ROUND(AVG(x)). Some reasons only appear once the sample is consulted: the samples are still loading, a column is not sampled, too few rows match.

Decision	Alternative	Why
When in doubt, go exact	Approximate as much as possible	A slow correct answer is better than a fast wrong one.
Forward the original SQL text	Rebuild SQL from the AST	Rebuilding can change a query in subtle ways.
Forward SQL that sqlparser cannot parse	Reject it	PostgreSQL is the judge of valid SQL: it runs it or returns its own error.
Separate intent (1) and planner (2)	One function	Real databases also separate parsing, analysis and planning; each question stays small.

Code: src/planner.rs, src/router.rs, src/database.rs.

3 · The sample

In plain words. The first time, the engine reads the whole table once and keeps a fair random selection of 100,000 rows, where every row has the same chance of being picked. It also keeps, for each row, its primary key, so it can recognise the row later when it changes.

Technical details: reservoir sampling. The table is read as a stream of unknown length. Reservoir sampling (Algorithm R) keeps a uniform sample of k rows in one pass and O(k) memory:

flowchart LR
    A(["row number i<br/>arrives"]) --> B{"reservoir<br/>full?"}
    B -- "no" --> C["store it"]
    B -- "yes" --> D["draw a random<br/>position j in 0..i"]
    D --> E{"j < k ?"}
    E -- "yes: probability k / i" --> F["replace the row<br/>in slot j"]
    E -- "no" --> G["skip it"]

Loading

Row i enters with probability k/i and later survives each row m with probability (m−1)/m, so every row ends up in the sample with the same probability, k/N. The same pass hashes every value into one HyperLogLog sketch per column, for COUNT(DISTINCT).

Decision	Alternative	Why
Reservoir sampling on the client	PostgreSQL's TABLESAMPLE SYSTEM	SYSTEM samples whole disk pages, so rows stored together are sampled together.
A fixed number of rows (100,000)	A percentage	The margin of error depends on n, not on the table size (error ∝ 1/√n): the reason 1,000 people are enough to poll a country. How to choose n: Two knobs.
Decide before reading a row whether to keep it	Read every row, then decide	After the first 100,000 rows, almost every row is skipped; this avoids copying 98% of the table into memory.
A portal (server-side cursor), 10,000 rows per round trip	One big SELECT into memory	Memory stays flat whatever the table size.
All tables in one read-only transaction	One transaction per table	Every table is read as of the same moment.
SET LOCAL synchronize_seqscans = off	The default	Otherwise a scan may start in the middle of the table and the same seed picks a different sample.
Store the sample by column	By row	See Performance.

Numeric columns (integer, bigint, numeric, real, double precision) become f64; text columns stay text. Dates, booleans and JSON are not sampled: queries on them go to PostgreSQL.

Code: src/sampler.rs.

4 · The math

In plain words. The average of a random sample lands close to the average of the whole table, and statistics says how close. Counting distinct values needs a different trick with hashes. For GROUP BY, the same estimate is made once per group, after checking that every group is well represented.

AVG: the Central Limit Theorem. The mean of a random sample is approximately normally distributed around the true mean:

estimate        = x̄                      (mean of the n matching sampled values)
standard error  = s / √n  × √(1 − n/N)    (s = sample standard deviation)
margin          = z × standard error       (z = 1.96 at 95% confidence)

z comes from the confidence level: 95% of a normal distribution lies within ±1.96 standard errors of its centre, 99% within ±2.576 (see Two knobs). √(1 − n/N) (finite population correction) shrinks the margin when the sample is a big part of the table, down to 0 when it is the whole table.

SUM and COUNT: totals. Each sampled row contributes its value (or 1 for COUNT) when it matches the WHERE clause, and 0 otherwise. The table total is N × the mean contribution, and its margin covers both the values and how many rows match. COUNT(*) without WHERE is exact: N is known.

COUNT(DISTINCT): HyperLogLog (Flajolet et al., 2007). A sample cannot count distinct values (customers repeat). Every value is hashed; the longest run of leading zero bits across 16,384 registers reveals how many distinct values were seen: ±1.6% (at 95%) in 16 KB per column, whatever the count. Its margin uses the same z as the rest.

GROUP BY. A sample can miss a rare group entirely, yet SQL would list it. So before estimating per group, two checks: the sample holds as many groups as the column's HyperLogLog distinct count (no group missing), and every group has at least 30 matching sampled rows. Up to 100 groups.

The engine refuses to estimate, and sends the query to PostgreSQL, when fewer than 30 sampled rows match (the CLT needs enough values), when COUNT(DISTINCT) is filtered or grouped (a sketch cannot filter), when a column is not sampled, or when there is nothing to average (SQL returns NULL).

WHERE on the sample. The WHERE clause is compiled once per query into a small Predicate tree: =, <>, <, <=, >, >= between a column and a constant, AND, OR, NOT, BETWEEN, IN. It follows SQL's three-valued logic (a comparison with NULL is unknown). Text only supports = and <>, because PostgreSQL orders text by language rules (collation).

Decision	Alternative	Why
Welford's algorithm	Sum of squares minus square of sum	The textbook formula returns a variance of −170.7 (impossible; true value 30) for 10⁹+4, 10⁹+7, 10⁹+13, 10⁹+16.
HyperLogLog with 2¹⁴ registers	2¹² (±3.2%) or 2¹⁶ (±0.8%)	16 KB per column for ±1.6% is a good balance.
GROUP BY on the uniform sample, with two safety checks	Stratified sampling (one sample per group)	Much simpler, and safe: when a group is too rare, the query goes to PostgreSQL. Stratification is on the roadmap.

Code: src/statistics.rs, src/hyperloglog.rs, src/predicate.rs, src/engine.rs.

5 · Output

An estimate must never look like an exact number. At the prompt, every approximate answer shows ≈, its margin, the confidence level, how much data it came from, and how fresh the sample is (live, synced 0.6 s ago). Over the network, the rows look like PostgreSQL's (numbers are numbers, so tools can use them), and the margin arrives as a NOTICE message that psql prints.

Code: src/format.rs.

---

Staying fresh: live changes

In plain words. A poll taken yesterday says nothing about today. So the engine subscribes to PostgreSQL's own record of changes: every INSERT, UPDATE and DELETE reaches the sample about a second later. In the lab, 100,000 new rows in the MEA region were reflected within 2.5 seconds: COUNT(*) went from 5,000,000 to exactly 5,100,000, MEA's row count to ≈ 342,975 ± 7,839 (true: 350,785) and its average price from ≈ 133 to ≈ 208.70 ± 3.53 (true: 208.84). Deleting them brought COUNT(*) back to exactly 5,000,000.

Technical details. PostgreSQL writes every change to its write-ahead log (WAL) before applying it. With wal_level = logical, a replication slot turns the WAL back into row changes, in commit order, and keeps them until the reader confirms it has them. The engine reads them with SQL functions and the test_decoding plugin that ships with PostgreSQL:

table public.sales: INSERT: id[bigint]:7 region[text]:'EU' price[numeric]:19.99
table public.sales: UPDATE: id[bigint]:7 region[text]:'NA' price[numeric]:21.00
table public.sales: DELETE: id[bigint]:7

sequenceDiagram
    autonumber
    participant App as Application
    participant PG as PostgreSQL
    participant Slot as Replication slot
    participant Sync as Sync task
    participant Disk as Snapshot file
    App->>PG: INSERT / UPDATE / DELETE
    PG->>Slot: the change, in commit order
    loop every second
        Sync->>Slot: peek at new changes
        Slot-->>Sync: BEGIN · changes · COMMIT
        Sync->>Sync: apply them to the sample (random pairing)
    end
    loop every 30 seconds, and on shutdown
        Sync->>Disk: save the samples and the position (LSN)
        Sync->>Slot: advance to the saved position
    end

Loading

Keeping the sample uniform. INSERT is the easy case: reservoir sampling was invented for growing streams, so a new row enters with probability k/N. DELETE is harder: deleting a sampled row leaves a hole, and filling it with any row would favour some rows. Random pairing (Gemulla, Lehner and Haas, A Dip in the Reservoir, VLDB 2006) pairs every deletion with a later insertion, using two counters of uncompensated deletions; the paper proves the sample stays uniform. A test checks it: after deleting half of a table and inserting as many rows, 20,000 times, every remaining row was sampled within ±10% of the expected rate. Until enough new rows arrive, a sample shrinks a little after deletions (it stays uniform; its margins are slightly wider). UPDATE changes values, not membership: a sampled row gets its new values.

Exactly once. Changes are peeked, not consumed. The slot is advanced only to a position that is already saved in the snapshot, so after a crash the engine reloads the snapshot and skips what it had applied. Nothing is lost and nothing counts twice.

The first scan and the stream must not overlap. The slot is created before the first scan, so no change can slip between the two. But a transaction that commits in that gap is both seen by the scan and present in the stream. The engine keeps the scan's MVCC snapshot (xmin:xmax:running) and skips the transactions it saw, exactly as PostgreSQL decides visibility, including transaction-id wraparound. This is not theoretical: with that filter disabled, the end-to-end test counted 200,301 or 200,302 rows in a 200,300-row table, in 3 runs out of 3.

Limits of following changes:

• A sketch cannot forget a value, so after an UPDATE or DELETE, COUNT(DISTINCT) goes to PostgreSQL until \resample rebuilds the sketches. INSERTs keep them exact.
• UPDATE and DELETE need a single-column primary key to find the row. A table without one follows INSERTs only; an UPDATE or DELETE on it marks the sample out of sync, and queries go to PostgreSQL until \resample.
• TRUNCATE empties the sample and the sketches (which stay exact).

Code: src/decoding.rs, src/changes.rs, src/sync.rs.

---

Fast restarts: snapshots

In plain words. Reading 5 million rows takes 12–18 seconds, and a billion rows would take hours. So the engine saves its samples to a file (6.8 MB for 100,000 rows of 7 columns) and, on restart, loads it in 0.2 seconds. The changes made while it was stopped are waiting in the replication slot, and are applied right after.

Technical details. The file (snapshots/aqp-engine.snapshot, in the compact binary format postcard) holds the samples, their sketches, the random pairing counters, and the WAL position of the last change applied. It is written to a temporary file and then renamed, so a crash never leaves half a file. On start, it is used only if it can be continued safely: same tables and sample size, the slot still exists, the slot has not moved past the saved position, and PostgreSQL has not discarded WAL the slot needed. Otherwise the engine rebuilds from scratch. The end-to-end test checks that a restart continues the file (it is not rewritten) and applies the changes made while the engine was down.

Code: src/snapshot.rs, src/sync.rs, src/store.rs.

---

Serving clients: the PostgreSQL protocol

In plain words. Any program that can talk to PostgreSQL can talk to the engine: psql, BI tools, scripts. It connects to port 6432 instead of 5432, and gets fast approximate answers where possible and PostgreSQL's exact answers everywhere else.

Technical details (built on pgwire):

• One PostgreSQL session per client, so transactions (BEGIN ... COMMIT) and settings stay separate between clients. Checked with psql: transactions, pass-through queries, PostgreSQL's own error messages.
• Approximate rows: estimates as float8 (counts as int8), group values as text, plus a NOTICE with the margin of error.
• Exact results pass through as text, with PostgreSQL's command tags (INSERT 0 3, BEGIN...). They are buffered in the engine before being sent, as psql also does.
• SHOW aqp_status; returns the engine's state as rows.
• The engine speaks the simple query protocol, the one psql uses. Drivers must use it too: psycopg2 does by default; JDBC needs preferQueryMode=simple. The extended protocol (prepared statements) is on the roadmap.

Code: src/server.rs, src/bin/server.rs.

---

Security

Concern	What the engine does
Clients logging in	SCRAM-SHA-256 with AQP_SERVER_USER / AQP_SERVER_PASSWORD (default aqp/aqp; the server warns about the default). The password never travels over the network.
Where it listens	127.0.0.1:6432 by default: only this machine.
Engine → PostgreSQL encryption	TLS, controlled by the URL's sslmode like psql: prefer (default) encrypts when the server offers it, require refuses to connect without it, disable never encrypts. The certificate is verified against the system's authorities, plus AQP_TLS_CA_FILE if set. Verified in the lab against a TLS-only PostgreSQL: encrypted with the CA file (ssl = t, TLS 1.2), refused without it.
What clients can do	Everything the engine's PostgreSQL user can do: exact queries pass through. Give that user only the privileges clients should have.
Following changes	Requires a PostgreSQL user with the REPLICATION attribute (or superuser).
SQL injection through names	Table and column names are always quoted ("name", with inner quotes doubled); values go as query parameters.

Not done yet: TLS between clients and the engine (put it behind a TLS proxy, or keep it on a private network), and per-client PostgreSQL users.

---

Performance and optimizations

Storing the sample by column (measured). The first version stored rows (Vec<Vec<Value>>): AVG(price) jumped through 100,000 separate rows of 168 bytes to read 8 bytes from each. Storing each column contiguously (Vec<Option<f64>>), as ClickHouse, DuckDB or Snowflake do, lets the CPU use almost every byte it loads (it reads memory in cache lines of 64–128 bytes). Same sample, identical estimates:

Layout	AVG(price)	COUNT(*) WHERE region = 'EU'	SUM(quantity) WHERE price > 100	Slowest query
By row	0.96 ms	5.87 ms	9.84 ms	10.48 ms
By column	1.08 ms	2.69 ms	3.04 ms	4.14 ms

GROUP BY without copying. Grouping first built a text key for every sampled row: 100,000 small memory allocations per query. Keys that borrow the text from the sample instead (GroupKey<'a>, a lifetime) avoid them. A GROUP BY region took 31 ms before (a single measurement) and 9–12 ms after, once warm.

Start-up. Reading the table once took 12–18 s; PostgreSQL alone needs 10.5 s of that just to read and convert the rows (COPY ... TO '/dev/null'), so the scan is bound by PostgreSQL on 2 CPUs. After the first time, the snapshot loads in 0.2 s.

Also: the WHERE clause is compiled once and evaluated into true/false marks shared by every aggregate; rows the reservoir skips are never copied; changes are applied in batches under one short lock, while queries keep reading.

---

What is supported, and what is not

Approximated:

• AVG(col), SUM(col), COUNT(*), COUNT(col), COUNT(DISTINCT col), several per query, with aliases.
• One table, optional WHERE (=, <>, <, <=, >, >=, AND, OR, NOT, BETWEEN, IN), optional GROUP BY of one column (or GROUP BY 1).
• Several tables sampled at once (AQP_TABLES=sales,events), each with its own sample, all read in the same transaction and followed through the same slot. A table smaller than the sample is held whole, and its answers are exact.

Sent to PostgreSQL (with the reason): anything that is not an aggregate query; JOIN, HAVING, sub-queries, UNION, WITH, ORDER BY, LIMIT, DISTINCT; aggregates inside expressions (ROUND(AVG(x)), SUM(x) * 1.19, AVG(price * quantity)); AVG(DISTINCT x), window functions, FILTER; COUNT(DISTINCT) with WHERE or GROUP BY, or after UPDATE/DELETE; conditions on columns that are not sampled, LIKE, IS NULL, functions; groups with fewer than 30 sampled rows, or missing from the sample; more than 100 groups.

Known limitations:

• Rare filters and rare groups get wide margins, or fall back to PostgreSQL. Stratified sampling would help (roadmap).
• Only the simple query protocol; no TLS between clients and the engine.
• Tested in a local lab (5 million rows, one machine), not under production load.

---

Running it in production

• Put it only in front of the features that aggregate, with their own connection string; see Where it fits. To shorten the hop further: run it on the same machine as the application (~0.1 ms), merge it into a connection pooler the stack already has (PgBouncer and similar add that hop anyway), or call the Rust library in-process (no hop; the routing decision takes 5.8 µs).

• PostgreSQL 13 or newer (for pg_current_snapshot()), with wal_level = logical to follow changes. Without it, the engine still works with a static sample and says so in \status.

• Replication slots keep WAL. While the engine is stopped, its slot keeps every change it has not read, on PostgreSQL's disk. Set max_slot_wal_keep_size (the lab uses 1 GB): past that limit PostgreSQL discards the old WAL, and the engine notices on its next start and rebuilds the samples. When you stop using the engine for good, drop its slot:

  docker compose exec postgres psql -U aqp -d aqp -c "SELECT pg_drop_replication_slot('aqp_engine')"

• Keep the first scan off the primary: point AQP_SAMPLE_DATABASE_URL at a replica. PostgreSQL 16+ can also follow changes on a standby.

• Monitor with SHOW aqp_status; (sync mode, last sync, last save, errors, per-table state).

• Crash safety: a crash loses nothing; the engine reloads the snapshot and replays the changes since.

• \resample (or restarting with the snapshot deleted) rebuilds the samples; the old ones keep answering meanwhile.

---

Testing

cargo test

99 tests, no database needed:

File	What it checks
intent_detection.rs	Aggregates found at any depth; exact queries and window functions stay exact.
planner.rs	Accepted shapes (including GROUP BY), and a reason for every rejected one.
router.rs	The route chosen from the SQL text.
predicate.rs	WHERE evaluation, BETWEEN, IN, SQL's NULL logic.
sampling.rs	Reservoir size, seeds, uniformity over 20,000 runs.
hyperloglog.rs	Accuracy from 5 to 1,000,000 distinct values.
estimation.rs	Welford, 95% and 99% coverage over 1,000 samples, margins that widen by the ratio of z-scores, totals, exact answers, NULLs, GROUP BY and its safety checks, every refusal.
decoding.rs	Reading test_decoding lines: quotes, types with spaces or brackets, key changes, TRUNCATE.
changes.rs	Random pairing stays uniform under deletes and inserts; updates, key changes, TRUNCATE, tables without keys.
sync.rs	WAL positions and the scan filter, including transaction-id wraparound.
snapshot.rs	A snapshot reads back identical.
server.rs	A real PostgreSQL client against the server: SCRAM login, wrong password, approximate answers, GROUP BY, SHOW aqp_status, errors that keep the connection alive.
format.rs	Number formatting and the output text.

One end-to-end test against the lab (it creates and removes the table aqp_live_test and the slot aqp_test_slot):

cargo test --release --test live_sync -- --ignored

It builds a sample of a 200,000-row table while another connection keeps inserting, applies 20,000 inserts, 30,000 deletes and 50,000 updates, checks that N equals count(*) exactly and that every sampled row exists with its current value, restarts the engine, changes the table while it is down, and checks again.

Tests that can fail. Deliberate bugs were introduced to prove the tests catch them: a z-score of 1.0 (coverage fell to 68.6%), a biased reservoir (uniformity failed), and a disabled scan filter (N off by 1–2 in 3 of 3 runs). The code was restored each time. cargo fmt --check and cargo clippy --all-targets pass with no warnings.

---

Project layout

aqp-engine/
├── Cargo.toml                 # package manifest and dependencies
├── rustfmt.toml               # code style for `cargo fmt`
├── docker-compose.yml         # the PostgreSQL lab (logical decoding on)
├── docker/init/01-seed.sh     # generates the fake sales data on the first start
├── src/
│   ├── main.rs                # the interactive engine (aqp> prompt)
│   ├── bin/server.rs          # the PostgreSQL-compatible server
│   ├── bin/benchmark.rs       # exact vs approximate, side by side
│   ├── bin/sample_size.rs     # margin, query time and memory for six sample sizes
│   ├── lib.rs                 # the list of modules
│   ├── config.rs              # settings from environment variables
│   ├── intent.rs              # does the query ask for an aggregate?
│   ├── planner.rs             # can the sample answer this shape?
│   ├── router.rs              # sample or PostgreSQL, and fallbacks
│   ├── database.rs            # connections: TLS, reconnection, exact queries
│   ├── sampler.rs             # the sample (columnar) and the first scan
│   ├── predicate.rs           # WHERE evaluation on the sample
│   ├── statistics.rs          # CLT estimates, margins and the confidence level
│   ├── hyperloglog.rs         # distinct counts
│   ├── engine.rs              # answers a query (and GROUP BY) from the sample
│   ├── decoding.rs            # reads PostgreSQL's change stream
│   ├── changes.rs             # applies changes (random pairing)
│   ├── sync.rs                # background task: load, follow changes, save
│   ├── snapshot.rs            # saving the samples to disk
│   ├── store.rs               # samples shared between queries and the sync task
│   ├── server.rs              # PostgreSQL protocol, SCRAM, per-client sessions
│   ├── status.rs              # \status and SHOW aqp_status
│   └── format.rs              # output text
├── tests/                     # one test file per module, plus the end-to-end test
├── scripts/                   # plot_sample_size.py draws the sample-size chart
└── docs/                      # the sample-size measurements (CSV) and chart (SVG)

---

Stack

Crate	Role
sqlparser	SQL text → AST
tokio	Async runtime: network, timers, background tasks
tokio-postgres + postgres-native-tls	PostgreSQL client, with TLS
pgwire	PostgreSQL's network protocol, server side, with SCRAM
serde + postcard	Saving the samples to disk
rand	Seedable random numbers for sampling
anyhow, tracing	Errors and logs

---

Roadmap

Step	Status
Parser, AST and intent detection	✅
Planner and router, with fallbacks	✅
Reservoir sampling, columnar sample, several tables	✅
CLT estimates, HyperLogLog, WHERE, GROUP BY	✅
Live INSERT/UPDATE/DELETE (logical decoding, random pairing, exactly once)	✅
Snapshots and background loading	✅
PostgreSQL protocol server, SCRAM, per-client sessions	✅
TLS to PostgreSQL, reconnection	✅
Benchmark lab and end-to-end tests	✅

Next, roughly by value:

• An analytics-only mode (AQP_PASSTHROUGH=false) that refuses non-aggregate queries with a clear message, so a transactional path pointed at the engine by mistake shows up at once.
• Stream pass-through results instead of collecting them first: the first row of a large exact result would arrive in about 1 ms instead of after the whole result (135 ms for 100,000 rows today).
• Stratified sampling for GROUP BY: one reservoir per group, so rare groups get tight margins.
• Aggregates inside expressions (SUM(x) * 1.19, ROUND(AVG(x), 2)): evaluate with the estimate and scale the margin.
• Extended query protocol, so every driver works with its defaults.
• TLS on the server side and per-client PostgreSQL users.
• Per-day sketches, so COUNT(DISTINCT) survives deletes that follow a time-based retention.
• The binary pgoutput plugin instead of test_decoding, and filtering the stream by table.
• Sampling through the primary-key index to avoid even the first full scan.

---

Glossary

Term	Meaning
AQP	Approximate Query Processing: answering from a summary of the data, with a known error.
AST	Abstract Syntax Tree: a query as a tree of meaningful nodes.
Aggregate	A function that turns many rows into one value: AVG, SUM, COUNT.
Sample (n) / table (N)	The rows kept in memory, chosen at random / all the rows.
Reservoir sampling	Keeps a uniform random sample of fixed size from a stream of unknown length, in one pass.
Random pairing	Keeps a reservoir sample uniform when rows are also deleted, by pairing deletions with later insertions.
Central Limit Theorem	The mean of a large enough random sample is approximately normal around the true mean: what gives estimates a computable margin.
Margin of error	estimate ± z standard errors. At 95% confidence (z = 1.96) it contains the true value in about 95% of samples.
Confidence level / z-score	How often the margin must contain the true value (95%, 99%...), and the number of standard errors that takes (1.96, 2.576...).
HyperLogLog	Estimates distinct counts from hashed values, in fixed memory.
WAL	Write-ahead log: PostgreSQL's record of every change, written before the change is applied.
LSN	Log sequence number: a position in the WAL, written like 16/B374D848.
Logical decoding / CDC	Turning the WAL back into row changes (Change Data Capture).
Replication slot	PostgreSQL's bookmark for a reader of changes: it keeps the changes until the reader confirms them.
MVCC snapshot	Which transactions a query can see (xmin:xmax:running).
Snapshot file	The engine's samples saved to disk, to restart without a full scan.
Columnar storage	Each column's values stored together, for fast scans of one column.
Three-valued logic	SQL's true, false and unknown (comparisons with NULL).
SCRAM	A password login where the password itself never crosses the network.
TLS	Encryption of a network connection (what https uses).
Portal	A server-side cursor: PostgreSQL hands out a query's rows in batches.
Proxy	A program between a client and a server that decides what to forward.

---

Getting started

Prerequisites

• Rust 1.88 or newer (rustup). The project uses the 2024 edition.
• Docker with Docker Compose, for the PostgreSQL lab.
• About 1.5 GB of free disk for the default 5 million rows.

1. Start PostgreSQL with fake data

docker compose up -d --wait

The first start generates 5 million sales rows inside PostgreSQL (about a minute), with logical decoding switched on so the engine can follow changes. PostgreSQL listens on port 5433 of your machine.

2. Try the interactive engine

cargo run --release

The aqp> prompt appears at once. The sample is built in the background (12–18 s the first time, then 0.2 s from the saved file); until it is ready, queries simply go to PostgreSQL.

aqp> SELECT AVG(price) AS avg_price, COUNT(*), SUM(amount) FROM sales WHERE region = 'EU'
 avg_price    ≈ 134.14 ± 1.17  (± 0.9%)
 COUNT(*)     ≈ 1,487,100 ± 14,024  (± 0.9%)
 SUM(amount)  ≈ 607,629,023.00 ± 9,930,949.05  (± 1.6%)

approximate · 95% confidence · 29,742 of 100,000 sampled rows matched · table: 5,000,000 rows · live, synced 0.6 s ago · 18.52 ms

Command	What it does
\exact <sql>	Runs the query on PostgreSQL, skipping the sample.
\explain <sql>	Shows which path the query would take and why, without running it.
\ast <sql>	Prints the syntax tree (AST) that sqlparser builds.
\status	Shows the confidence level and sample size, each sample with its memory, how fresh they are, and the last save.
\resample	Rebuilds the samples from scratch in the background (the current ones keep answering).
\q	Saves the samples and quits.

3. Or run the server and connect with psql

cargo run --release --bin server

psql "host=127.0.0.1 port=6432 user=aqp dbname=aqp"

The default password is aqp; set AQP_SERVER_PASSWORD to choose one. SHOW aqp_status; describes the samples. Ctrl+C saves the samples and stops the server.

4. Benchmark and tests

cargo run --release --bin benchmark

cargo test

The 99 tests need no database. One more end-to-end test runs against the lab (see Testing).

To measure the sample sizes again and redraw the chart (the script needs matplotlib: pip install matplotlib):

AQP_SEED=42 cargo run --release --bin sample_size

python3 scripts/plot_sample_size.py

The first builds a sample at each of six sizes (every build scans the table once) and writes docs/sample-size.csv; the second turns it into docs/sample-size-light.svg and docs/sample-size-dark.svg. Nothing is left behind in PostgreSQL: these samples do not follow changes, so they create no replication slot.

5. Stop or delete the lab

docker compose stop

docker compose down -v

The first stops PostgreSQL and keeps the data; the second deletes it.

---

Configuration

Everything has a default that matches the lab.

Variable	Default	Meaning
DATABASE_URL	postgres://aqp:aqp@localhost:5433/aqp	PostgreSQL for exact queries. Add ?sslmode=require to insist on TLS.
AQP_SAMPLE_DATABASE_URL	same as DATABASE_URL	PostgreSQL the sample is read from and whose changes are followed (e.g. a replica).
AQP_TLS_CA_FILE	–	Extra certificate authority to trust (PEM file).
AQP_TABLES	sales	Tables to sample, comma-separated, without schema.
AQP_SAMPLE_SIZE	100000	Rows per sample: the knob that buys precision. Margins shrink with √n; query time and memory grow with n. See Two knobs.
AQP_CONFIDENCE	0.95	Confidence level of every margin: 0.90, 0.95, 0.98, 0.99 or 0.999 (99 and 99% also work). Higher means a wider margin, at no other cost.
AQP_SEED	from the clock	Random seed; the same seed on the same data picks the same sample.
AQP_LIVE_SYNC	true	Follow changes through logical decoding.
AQP_SLOT_NAME	aqp_engine	Replication slot name.
AQP_SYNC_INTERVAL_MS	1000	How often changes are read.
AQP_SNAPSHOT_FILE	snapshots/aqp-engine.snapshot	Where the samples are saved.
AQP_SNAPSHOT_INTERVAL_SECS	30	How often the samples are saved.
AQP_SERVER_ADDRESS	127.0.0.1:6432	Where the server listens.
AQP_SERVER_USER / AQP_SERVER_PASSWORD	aqp / aqp	Login for clients of the server.
RUST_LOG	off	Logs, e.g. RUST_LOG=aqp_engine=info.

---

The lab (Docker)

docker-compose.yml starts PostgreSQL 17 with wal_level=logical and max_slot_wal_keep_size=1GB. On the first start, docker/init/01-seed.sh creates sales and fills it inside PostgreSQL with generate_series and random():

Column	How it is generated
id	1, 2, 3... (primary key)
sold_at	a random moment in the last year (not sampled: queries on it go to PostgreSQL)
region	NA 35%, EU 30%, APAC 20%, LATAM 10%, MEA 5%
category	electronics 30%, home 25%, clothing 20%, sports 15%, books 10%
customer_id	1 to 200,000, skewed: a few customers buy a lot
quantity	1 to 10, mostly 1 to 3
price	25 to about 410, long-tailed
amount	price × quantity

The data is deliberately skewed, like real data. To change the row count (about 120 MB per million rows):

docker compose down -v

SALES_ROWS=10000000 docker compose up -d --wait