Skip to content

Make Postgres work on real schemas - #159

Draft
mmkal wants to merge 29 commits into
mainfrom
pg-real-schemas
Draft

mmkal wants to merge 29 commits into
mainfrom
pg-real-schemas

Conversation

@mmkal

@mmkal mmkal commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #156. Fixes most of the Postgres "Known gaps" that guide listed: real-world schemas now round-trip, generated wrappers have the same shapes as on SQLite, and pg errors get a real kind.

Schemas. serial/bigserial, identity columns, enums, domains, stored generated columns, functions, triggers, materialized views, extensions and non-public schemas now work in draft, check and generate. Before, generate failed on a serial pk (relation "posts_id_seq" does not exist), identity columns were re-added in every draft, and functions/triggers were redrafted every time.

Wrappers. Same queries.sql, generated for Postgres:

// before                                          // after (identical to SQLite)
findPostBySlug(client, {slug}): Promise<Result[]>  findPostBySlug(client, {slug}): Promise<Result | null>
insertPostReturningId(client, params) // no row       insertPostReturningId(client, params): Promise<Result>
updateTitle(client, {title, slug})                 updateTitle(client, {title}, {slug})  // data, params
params: {slug: string | null}                      params: {slug: string}

Errors.

try {
  await insertPost(client, {slug: 'taken', title: 'Hi', body: null});
} catch (error) {
  error.kind; // before: 'unknown'. after: 'unique_violation' (SQLSTATE 23505)
}
Change Purpose
packages/pg/src/impl/schema.ts: renderSchema replaces renderCanonicalSchema The hand-rolled pg_catalog renderer only knew public tables/columns/constraints/indexes/views. Rendering with the vendored migra (empty inspector → db) means anything diffSchema can diff, it can also render. omitTables keeps excludedTables working.
packages/pg/src/vendor/{migra,schemainspect}/* Bugs found by the round-trip spec: domains created after the tables using them, create domain missing ;, domain over an enum created twice (and its columns mistaken for enum columns, which crashed inspection when the schema had any view), column collate dropped from create table. Each marked with a sqlfu: comment.
packages/pg/src/impl/query-shape.ts (new), packages/pg/src/impl/typegen.ts The shared wrapper generator was right; the pg analyzer fed it wrong inputs. This reads the pgsql-ast-parser AST plus catalog facts (unique indexes, column nullability, aggregate/set-returning functions) to set multipleRowsResult, returning, data params and param nullability the way SQLite does. A with clause in front of DML is looked through, so with ... update is still an update.
packages/sqlfu/src/errors.ts, packages/sqlfu/src/adapter-errors.ts classifyPostgresError maps SQLSTATE → kind, picked by client.system. mapSqliteDriverError → mapDriverError. SQLite adapters unchanged.
packages/pg/test/schema-round-trip.test.ts Spec: render then diff returns nothing, for each shape above plus the ones that already worked. Known-broken shapes are test.fails.
packages/pg/test/generate-parity.test.ts Spec: one project generated with both dialects must produce identical wrapper signatures. SQLite's output pinned as an inline snapshot.
packages/pg/test/client-errors.test.ts Spec: each kind produced by real Postgres (including real lock conflicts for transient), plus an unknown case.
packages/pg/test/fixtures/typegen/*.md, typegen-fixture-runner.ts Fixtures now record multipleRowsResult, returning and data, so wrapper-shape regressions show up in the analyzer fixtures. New result-shape.md for pg-only rules.
packages/sqlfu/docs/guides/postgres.md, docs/errors.md, guides/pgkit.md, READMEs Known gaps now lists only what's still broken. Runtime example uses Result | null. Kind → SQLite code / SQLSTATE table in Errors. Dropped "use uuid ids for now".
tasks/complete/2026-10-29-pg-real-schemas.md, tasks/pg.md Spec, decisions and leftovers per part. Step three ticked.
Fixture output before/after

dml-returning.md, insert into posts (title) values ($1) returning id, title, draft:

 queryType: Insert
+returning: true
+multipleRowsResult: false
 columns: ...
 parameters:
-  - {name: $1, tsType: string, notNull: false}
+  - {name: $1, tsType: string, notNull: true}

Every fixture gains multipleRowsResult; selects with limit 1, no from, an ungrouped aggregate, or a unique-key lookup flip to false. Params compared with = flip to notNull: true.

Decisions to review (made without you; details in the task file):

  • Wrapper rules copy SQLite's, warts included, so code ports between dialects. E.g. DML+RETURNING is always one row; params are not-null unless tested with is null or written to a nullable column.
  • Unique-key single-row detection checks every unique index (SQLite/typesql only spots tables with exactly one unique column).
  • transient = serialization failure, deadlock, lock not available. Connection errors stay unknown.
  • Rendered schema uses migra's statement style (same as pg drafts). Grants, comments and the default public schema/plpgsql extension stay out.

Still broken (in Known gaps and tasks/pg.md):

  • migra creates functions and composite types after tables, so a default/check calling a user function, or a composite-type column, fails.
  • bigint/numeric typed as numbers, but pg returns strings by default. cast(...) drops not-null.

Coordinating session: local_d8095ac3-f355-44e4-a174-578140e5095e (Postgres support status)
Part sessions: local_ed3d974f-6104-480c-859b-ff1425e11662 (renderer), local_ca9de0dd-5a56-41ce-8314-da6da8e927c1 (typegen), local_2c739eba-ccbe-4762-9e0c-c52d1a8c1946 (errors)

🤖 Generated with Claude Code

Package size — packed 288.3 kB (+496 B, +0.2%)

Package size

main this PR Δ
packed 287.8 kB 288.3 kB +0.2%
unpacked 1.16 MB 1.16 MB +0.1%
files 226 226 0

dist/vendor/*.js bundles

main this PR Δ
vendor/sha256.js 4.3 kB 4.3 kB 0
vendor/sql-formatter/*.js 94.5 kB 94.5 kB 0
vendor/sqlfu-sqlite-parser/*.js 17.2 kB 17.2 kB 0
vendor/standard-schema/*.js 2.8 kB 2.8 kB 0
vendor/typesql/*.js 135.2 kB 135.2 kB 0

Measured with npm pack --dry-run --json on sqlfu (0.0.0-dev on main vs 0.0.0-dev on this PR).

mmkal and others added 8 commits October 29, 2026 08:45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Postgres guide walks a pg project through draft, migrate, generate and
runtime use, verified end to end against the docker-compose Postgres in
packages/pg/test. It lists the gaps found while verifying: the simplified
schema reader in packages/pg/src/impl/schema.ts drops serial/identity,
enums, domains, functions, triggers, matviews, extensions, non-public
schemas and generated columns, and pg typegen differs from SQLite for
limit 1, returning, and nullable params.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The docs called Postgres a runtime adapter with toolchain docs 'still in
progress'. Postgres has a working dialect (draft, migrate, check, generate,
format, UI), so: README documents the dialect config key and points
limitations at the guide's known gaps; adapters, errors, formatter, lint
plugin and schema-diff pages say what's true for pg today (including that pg
errors are always kind 'unknown'); the intro blog post links the guide;
@sqlfu/pg gets a README so its npm page isn't empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tasks/pg.md was stale (step one 'in review', step two unticked though
PRs #87-#89 merged). It now has a current status summary and a step
three checklist of the bugs found while verifying the Postgres guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main added a celld guide alongside the Postgres guides here. Both sides
appended to the Guides description in sync-docs.mjs; kept both and
regenerated llms.txt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One PR for the three Postgres fixes found while documenting pg support:
schema renderer, typegen parity, error classification. Stacked on pg-docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pkg-pr-new

pkg-pr-new Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/sqlfu@159

commit: 76b8f13

mmkal and others added 3 commits October 29, 2026 13:22
createNodePostgresClient returns kind 'unknown' for every Postgres error
except syntax errors (which only match via the SQLite message fallback).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task file for replacing @sqlfu/pg's hand-rolled pg_catalog renderer
with the vendored migra, so materializeSchemaSql/extractSchemaFromClient
round-trip real Postgres schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
materializeSchemaSql/extractSchemaFromClient output, diffed against the
original schema in both directions, should be empty. Fails today for
serial, identity, enums, domains, generated columns, functions,
triggers, matviews, extensions and non-public schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mmkal mmkal mentioned this pull request Oct 29, 2026
mmkal added a commit that referenced this pull request Oct 29, 2026
Postgres works today via `@sqlfu/pg`, but the docs called it a runtime
adapter with "toolchain docs still in progress", and the `dialect`
config key wasn't documented anywhere. This adds two pages to Guides and
fixes the Postgres wording elsewhere.

New docs pages:

-
**[Postgres](https://github.com/iterate/sqlfu/blob/main/packages/sqlfu/docs/guides/postgres.md)**:
install, config, `draft` → `migrate` → `generate`, runtime use, how it
differs from SQLite, and **Known gaps**.
- **[Coming from
pgkit](https://github.com/iterate/sqlfu/blob/main/packages/sqlfu/docs/guides/pgkit.md)**:
pgkit packages and migrator commands mapped to sqlfu, plus what doesn't
carry over. It says plainly that sqlfu is not a drop-in replacement.

```ts
import {pgDialect} from '@sqlfu/pg';
import {createNodePostgresClient, defineConfig} from 'sqlfu';
import {Pool} from 'pg';

export default defineConfig({
  dialect: pgDialect({adminUrl: process.env.DATABASE_URL!}),
  db: () => {
    const pool = new Pool({connectionString: process.env.DATABASE_URL});
    return {client: createNodePostgresClient(pool), [Symbol.asyncDispose]: () => pool.end()};
  },
  definitions: './definitions.sql',
  migrations: './migrations',
  queries: './sql',
});
```

I ran the guide end to end against the docker-compose Postgres in
`packages/pg/test`. That turned up real bugs. They're **documented, not
fixed**, in the guide's Known gaps and in `tasks/pg.md` step three:

- `serial` breaks `generate` (`relation "posts_id_seq" does not exist`).
Identity columns get re-added by every later `draft`. Enums, domains,
functions, triggers, matviews, extensions and non-`public` schemas
either error or get redrafted every time. The cause is the simplified
schema renderer in `packages/pg/src/impl/schema.ts`. The guide uses
`uuid default gen_random_uuid()` ids because those work.
- pg typegen differs from SQLite: `limit 1` returns an array, `insert
... returning` wrappers return no rows, and `= :param` params are typed
nullable.
- pg errors are always `kind: 'unknown'`.

Fixes for these are in progress in #159.

| Change | Purpose |
| --- | --- |
| `packages/sqlfu/docs/guides/postgres.md` | Main entry point for
Postgres users. Snippets are copied from a project that ran end to end.
Known gaps list what's broken today, so nobody adopts it blind. |
| `packages/sqlfu/docs/guides/pgkit.md` | Answers "is this a pgkit
replacement?" (no), and gives pgkit users a command-by-command map. |
| `website/astro.config.mjs`, `website/scripts/sync-docs.mjs`,
`website/scripts/sync-llms.mjs`, `website/public/llms.txt`,
`docs/guides.md`, `docs/adapters.md` guide table | Puts both pages in
the Guides sidebar, guide tables and llms.txt (as guides, not
"Optional"). |
| `packages/sqlfu/README.md` (+ regenerated root `README.md`) | Intro
mentions Postgres. `dialect` is documented under Configuration.
Limitations point at Known gaps instead of "docs in progress". Errors
and prior-art lines are corrected. |
| `docs/adapters.md`, `docs/errors.md`, `docs/formatter.md`,
`docs/lint-plugin.md`, `docs/schema-diff-model.md` | Each says what's
true for pg now. `sqlfu format` follows the config, but the lint rule
needs `{dialect: 'postgresql'}`. pg errors aren't classified. The SQLite
diff engine isn't the pg one. |
| `website/src/content/blog/introducing-sqlfu.md` | The "what it is not
for" line said the pg story needed docs. It now links the guide (same
line PR #121 last touched). |
| `packages/pg/README.md`, `packages/pg/package.json` description |
`@sqlfu/pg`'s npm page was empty. |
| `tasks/pg.md`, `tasks/complete/2026-10-29-pg-docs.md` | `tasks/pg.md`
still said step one was in review with step two unticked. It now has a
current status and a step-three bug list. |

Verified: `astro build`, `node --test website/test/`, and the rendered
`dist/` links for the new pages.

Claude Code session: "Postgres support status"
(`local_d8095ac3-f355-44e4-a174-578140e5095e`, CLI
`403039a7-59d0-4757-851e-4d2a4e778da2`)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- package-size:start -->
<details>
<summary>Package size — packed 280.7 kB (+142 B, 0)</summary>

## Package size

|  | main | this PR | Δ |
| - | - | - | - |
| packed | 280.6 kB | 280.7 kB | 0 |
| unpacked | 1.12 MB | 1.12 MB | 0 |
| files | 226 | 226 | 0 |

### `dist/vendor/*.js` bundles

|  | main | this PR | Δ |
| - | - | - | - |
| `vendor/sha256.js` | 4.3 kB | 4.3 kB | 0 |
| `vendor/sql-formatter/*.js` | 94.5 kB | 94.5 kB | 0 |
| `vendor/sqlfu-sqlite-parser/*.js` | 17.2 kB | 17.2 kB | 0 |
| `vendor/standard-schema/*.js` | 2.8 kB | 2.8 kB | 0 |
| `vendor/typesql/*.js` | 134.9 kB | 134.9 kB | 0 |

_Measured with `npm pack --dry-run --json` on `sqlfu` (0.0.0-dev on main
vs 0.0.0-dev on this PR)._

</details>
<!-- package-size:end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
pg-docs (#156) was squash-merged to main; this drops its commits from #159's diff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mmkal
mmkal changed the base branch from pg-docs to main October 29, 2026 13:25
mmkal and others added 8 commits October 29, 2026 13:25
createNodePostgresClient errors now get a real SqlfuError.kind. The
classifier reads the SQLSTATE from the pg error's .code and is picked by
client.system, so the SQLite adapters are unchanged.
mapSqliteDriverError is renamed mapDriverError since it now covers both.

Removes the Errors note from the Postgres guide's Known gaps, adds a
kind -> SQLite / SQLSTATE table to docs/errors.md, and ticks the
step-three item in tasks/pg.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task file for step three item two of tasks/pg.md: limit 1 / FROM-less /
aggregate selects, DML RETURNING, update data split and where-param
nullability should generate the same wrappers on pg as on SQLite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	packages/sqlfu/README.md
#	packages/sqlfu/docs/errors.md
#	packages/sqlfu/docs/guides/postgres.md
#	tasks/pg.md
#160 was closed when its base branch was deleted; #163 replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generates one project with SQLite and with pgDialect and asserts the
wrapper signatures and namespace types match. All six cases fail today:
limit 1 / FROM-less / aggregate selects return arrays, RETURNING wrappers
call client.run, update SET params aren't split into data, and where
params are typed nullable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The error work now ships in the shared PR #159 rather than its own PR,
so its spec and decisions live in the "Error classification" section of
tasks/pg-real-schemas.md. tasks/pg.md keeps only the ticked checkbox;
the shared status-summary line is left for whoever finishes last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mmkal and others added 5 commits October 29, 2026 13:32
materializeSchemaSql and extractSchemaFromClient now render a schema as
migra's diff from an empty inspector to the inspected database, replacing
the hand-rolled pg_catalog renderer. diffSchema runs the same migra, so a
rendered schema diffs as identical to its source: serial, identity, enums,
domains, generated columns, functions, triggers, matviews, extensions and
non-public schemas all round-trip now.

Vendored migra/schemainspect fixes found on the way, each marked sqlfu::
- new domains are created before tables, and create domain ends with ;
- the enums query no longer returns domains over enums
- create table keeps column collations

Docs: Postgres known gaps shrink to the remaining migra ordering bug
(functions/composite types created after the tables using them), ICU
collations, and comments/grants. tasks/pg.md step three item ticked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	packages/sqlfu/README.md
#	packages/sqlfu/docs/guides/pgkit.md
#	packages/sqlfu/docs/guides/postgres.md
#	tasks/pg.md
New packages/pg/src/impl/query-shape.ts reads the pgsql-ast-parser AST
plus catalog facts (unique indexes, column nullability, aggregate and
set-returning functions) and fills the descriptor fields the shared
wrapper generator keys off:

- multipleRowsResult: false for limit 1, FROM-less selects, aggregates
  without group by, and lookups covering a unique index. DML is always
  single-row, as in typesql. Set-returning functions stay multi-row.
- params: not-null unless tested with is [not] null or inserted/set into
  a nullable column.
- data: update SET params, when they come before every other param.

returning is set for DML that has result columns, so RETURNING wrappers
call client.all and return the row instead of client.run.

Analyzer fixtures now record multipleRowsResult (plus returning/data when
set); new result-shape.md covers the pg-only rules. The Postgres guide's
Known gaps list the differences that remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Folds tasks/complete/2026-10-29-pg-lossless-schema.md into the
"Schema renderer" section of tasks/pg-real-schemas.md, points the
tasks/pg.md checkbox at PR #159, and restores the shared pg.md status
summary (its owner updates it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves the typegen-parity spec into the shared tasks/pg-real-schemas.md
(Typegen parity section) and ticks its tasks/pg.md checkbox.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7c7e176 meant to fold tasks/complete/2026-10-29-pg-lossless-schema.md
into the shared task's "Schema renderer" section but only deleted it.
This restores that spec (condensed), ticks the section, points tasks/pg.md
at #159 instead of the closed #161, and moves the task to complete now
that all three sections are done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mmkal
mmkal marked this pull request as ready for review October 29, 2026 18:11
@cursor

cursor Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes core Postgres schema materialization, typegen descriptors, and error handling paths used by draft/migrate/generate; broad test coverage limits regressions but migra ordering and AST fallbacks can still surprise edge schemas.

Overview
Postgres draft/check/generate no longer rely on a partial hand-rolled pg_catalog renderer. materializeSchemaSql and extractSchemaFromClient now emit DDL by inspecting the database with vendored schemainspect and diffing against an empty schema via migra, aligned with diffSchema. excludedTables still strips named tables and attached objects. Vendored migra/schemainspect fixes cover domain ordering, domain ;, enum-vs-domain query, and column collations in create table.

sqlfu generate on Postgres now feeds the shared wrapper generator the same descriptor signals SQLite uses: query-shape.ts parses SQL with pgsql-ast-parser plus catalog facts (unique keys, nullability, aggregates, set-returning functions) to set multipleRowsResult, returning, update data vs params, and param notNull. A SQLite vs pg parity test locks wrapper signatures.

Runtime errors from createNodePostgresClient map SQLSTATE to SqlfuError.kind (same vocabulary as SQLite) via mapDriverError, with integration tests against real Postgres.

Docs and Known gaps shrink to what’s still broken (e.g. functions/composite types created after tables, bigint/numeric string mismatch).

Reviewed by Cursor Bugbot for commit 0f0b852. Bugbot is set up for automated code reviews on this repo. Configure here.

@mmkal
mmkal marked this pull request as draft October 29, 2026 18:19

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0f0b852. Configure here.

Comment thread packages/pg/src/impl/query-shape.ts Resolved
mmkal and others added 2 commits October 29, 2026 18:28
schemainspect's relations query tagged a column `is_enum` when its type had
`typcategory = 'E'`. A domain over an enum has that category too, but the
enums query (already fixed to `typtype = 'e'`) doesn't list the domain. So
`load_deps` looked up the column's enum, got `undefined`, and threw
`Cannot read properties of undefined (reading 'signature')`. That walk only
runs when the schema has a dependency row, such as any view, which is why
the existing round-trip case passed. That case now includes a view, and
both relations queries use `typtype = 'e'`.

Reported by Cursor Bugbot on #159.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clause

`with ... insert` and `with ... update` were classified as selects, because
`classifyQuery` only looks at the first keyword. `findDataParams` and
`findNullableParams` also only checked the top-level statement. So such an
update never split its SET values into `data`, an insert never marked params
for nullable columns as nullable, and both came out as `Result | null`
selects instead of DML returning one row.

`query-shape.ts` now takes the statement after any `with` clause and uses it
for the query type, the data split, and the nullable-target checks. The
`is null` scan still covers the whole query. `classifyQuery` is only the
fallback when `pgsql-ast-parser` can't parse the query.

Reported by Cursor Bugbot on #159.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant