Skip to content

Render Postgres schemas losslessly with vendored migra - #164

Closed
mmkal wants to merge 10 commits into
mainfrom
pg-lossless-schema
Closed

mmkal wants to merge 10 commits into
mainfrom
pg-lossless-schema

Conversation

@mmkal

@mmkal mmkal commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Replaces #161, which GitHub closed when its base (pg-docs, #156) merged.

draft, check and generate re-read Postgres schemas as SQL text, then diff that text. The text came from a hand-rolled pg_catalog renderer that only knew public tables, columns, constraints, indexes and views. Everything else was lost, so most real schemas broke: serial made generate fail, identity columns came back in every draft, enums and domains didn't replay.

Now the text is migra's diff from an empty database to yours (what pgkit's migrator does for definitions.sql). diffSchema already uses the same migra, so whatever it can diff, the renderer can render.

Round-trips after this PR (materializeSchemaSql, then diffSchema both ways, empty):

serial/bigserial, identity, enums, domains (incl. over enums), stored generated columns, functions, triggers, materialized views, extensions, non-public schemas, column collations. Plus 30-odd shapes probed by hand (partitions, RLS, inheritance, exclusion constraints, setof functions, ...).

Rendered schema, before / after

create type post_status as enum ('draft', 'published');
create table posts (id serial primary key, slug text not null unique, status post_status not null default 'draft');

Before (no enum, no sequence, constraint names dropped):

create table "posts" (
  "id" integer not null default nextval('posts_id_seq'::regclass),
  "slug" text not null,
  "status" post_status not null default 'draft'::post_status,
  PRIMARY KEY (id),
  UNIQUE (slug)
);

After (same style as Postgres drafts):

create type "public"."post_status" as enum ('draft', 'published');

create sequence "public"."posts_id_seq";

create table "public"."posts" (
    "id" integer not null default nextval('posts_id_seq'::regclass),
    "slug" text not null,
    "status" post_status not null default 'draft'::post_status
      );

alter sequence "public"."posts_id_seq" owned by "public"."posts"."id";

CREATE UNIQUE INDEX posts_pkey ON public.posts USING btree (id);

CREATE UNIQUE INDEX posts_slug_key ON public.posts USING btree (slug);

alter table "public"."posts" add constraint "posts_pkey" PRIMARY KEY using index "posts_pkey";

alter table "public"."posts" add constraint "posts_slug_key" UNIQUE using index "posts_slug_key";

This is what the UI's live-schema pane shows now.

Still broken

migra creates every function and composite type after the tables. So a table whose default, check or generated column calls a user function, or that has a composite-type column, fails, in drafts too. Two test.fails cases pin it. It's a migra draft-ordering change that reshuffles fixture output, so it's left for its own PR (listed in tasks/pg.md).

Also listed in the guide: custom ICU collations lose locale/deterministic; comments and grants are never drafted.

Change Purpose
packages/pg/src/impl/schema.ts: renderCanonicalSchema + its pg_catalog queries replaced by renderSchema (migra from PostgreSQL.empty() to the inspected db) One schema model for both rendering and diffing, so a rendering always diffs as identical to its source.
omitTables in the same file excludedTables (e.g. sqlfu_migrations) drops the table plus its indexes, constraints, triggers, policies and owned sequences from the inspector.
Leave out the public schema and plpgsql extension Every new database has them; create schema if not exists "public" was noise in the UI pane.
vendor/migra/migra.ts, changes.ts: create new domains before tables migra created domains after the tables using them (type "email" does not exist). Modified domains keep the old drop-then-create spot.
vendor/schemainspect/pg/obj.ts: create domain ends with ; Without it the domain merged into the next statement.
vendor/schemainspect/queries.ts: enums query uses typtype = 'e' typcategory = 'E' also matched domains over enums, so they were created twice.
vendor/schemainspect/inspected.ts: collation in ColumnInfo.creation_clause create table dropped collate "C"; only add column kept it.
packages/pg/test/schema-round-trip.test.ts The spec: each shape rendered, then diffed both ways. Also extractSchemaFromClient on a live db and excludedTables.
docs/guides/postgres.md, guides/pgkit.md, packages/sqlfu/README.md Known gaps shrink to what's still true; drop "use uuid or text ids for now" and the pgkit "schema coverage" caveat.
tasks/pg.md, tasks/complete/2026-10-29-pg-lossless-schema.md Step three item ticked; function/composite-type ordering added as the next item. Task file has the probe results and decisions.

Tested: @sqlfu/pg typecheck + full suite (postgres:16 from packages/pg/test/docker-compose.yml), and packages/ui/test/pg-studio.spec.ts (3/3).

Session: "Fix Postgres schema renderer for real schemas" (Claude Code desktop local_ed3d974f-6104-480c-859b-ff1425e11662, CLI 7f20d0b1-4a1e-4bfc-9594-63bff2958ec5)

🤖 Generated with Claude Code

Package size — packed 280.7 kB (-2 B, 0)

Package size

main this PR Δ
packed 280.7 kB 280.7 kB 0
unpacked 1.12 MB 1.12 MB 0
files 226 226 0

dist/vendor/*.js bundles

main this PR Δ
vendor/sha256.js 4.3 kB 4.3 kB 0
vendor/sql-formatter/*.js 94.5 kB 94.5 kB 0
vendor/sqlfu-sqlite-parser/*.js 17.2 kB 17.2 kB 0
vendor/standard-schema/*.js 2.8 kB 2.8 kB 0
vendor/typesql/*.js 134.9 kB 134.9 kB 0

Measured with npm pack --dry-run --json on sqlfu (0.0.0-dev on main vs 0.0.0-dev on this PR).

mmkal and others added 10 commits October 29, 2026 08:45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Postgres guide walks a pg project through draft, migrate, generate and
runtime use, verified end to end against the docker-compose Postgres in
packages/pg/test. It lists the gaps found while verifying: the simplified
schema reader in packages/pg/src/impl/schema.ts drops serial/identity,
enums, domains, functions, triggers, matviews, extensions, non-public
schemas and generated columns, and pg typegen differs from SQLite for
limit 1, returning, and nullable params.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The docs called Postgres a runtime adapter with toolchain docs 'still in
progress'. Postgres has a working dialect (draft, migrate, check, generate,
format, UI), so: README documents the dialect config key and points
limitations at the guide's known gaps; adapters, errors, formatter, lint
plugin and schema-diff pages say what's true for pg today (including that pg
errors are always kind 'unknown'); the intro blog post links the guide;
@sqlfu/pg gets a README so its npm page isn't empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tasks/pg.md was stale (step one 'in review', step two unticked though
PRs #87-#89 merged). It now has a current status summary and a step
three checklist of the bugs found while verifying the Postgres guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main added a celld guide alongside the Postgres guides here. Both sides
appended to the Guides description in sync-docs.mjs; kept both and
regenerated llms.txt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task file for replacing @sqlfu/pg's hand-rolled pg_catalog renderer
with the vendored migra, so materializeSchemaSql/extractSchemaFromClient
round-trip real Postgres schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
materializeSchemaSql/extractSchemaFromClient output, diffed against the
original schema in both directions, should be empty. Fails today for
serial, identity, enums, domains, generated columns, functions,
triggers, matviews, extensions and non-public schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
materializeSchemaSql and extractSchemaFromClient now render a schema as
migra's diff from an empty inspector to the inspected database, replacing
the hand-rolled pg_catalog renderer. diffSchema runs the same migra, so a
rendered schema diffs as identical to its source: serial, identity, enums,
domains, generated columns, functions, triggers, matviews, extensions and
non-public schemas all round-trip now.

Vendored migra/schemainspect fixes found on the way, each marked sqlfu::
- new domains are created before tables, and create domain ends with ;
- the enums query no longer returns domains over enums
- create table keeps column collations

Docs: Postgres known gaps shrink to the remaining migra ordering bug
(functions/composite types created after the tables using them), ICU
collations, and comments/grants. tasks/pg.md step three item ticked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	README.md
#	packages/sqlfu/README.md
#	packages/sqlfu/docs/guides/pgkit.md
#	packages/sqlfu/docs/guides/postgres.md
#	tasks/pg.md
@pkg-pr-new

pkg-pr-new Bot commented Oct 29, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/sqlfu@164

commit: 8622ab6

@mmkal

mmkal commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author

🤖 Folded into #159 (shared "Make Postgres work on real schemas" PR, branch pg-real-schemas), which now has these commits.

@mmkal mmkal closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant