Skip to content

Render Postgres schemas losslessly with vendored migra - #161

Closed
mmkal wants to merge 2 commits into
pg-docsfrom
pg-lossless-schema
Closed

mmkal wants to merge 2 commits into
pg-docsfrom
pg-lossless-schema

Conversation

@mmkal

@mmkal mmkal commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #156 (pg-docs), because it edits that PR's "Known gaps" table.

draft, check and generate re-read Postgres schemas as SQL text, then diff that text. The text came from a hand-rolled pg_catalog renderer that only knew public tables, columns, constraints, indexes and views. Everything else was lost, so most real schemas broke: serial made generate fail, identity columns came back in every draft, enums and domains didn't replay.

Now the text is migra's diff from an empty database to yours (what pgkit's migrator does for definitions.sql). diffSchema already uses the same migra, so whatever it can diff, the renderer can render.

Round-trips after this PR (materializeSchemaSql, then diffSchema both ways, empty):

serial/bigserial, identity, enums, domains (incl. over enums), stored generated columns, functions, triggers, materialized views, extensions, non-public schemas, column collations. Plus 30-odd shapes probed by hand (partitions, RLS, inheritance, exclusion constraints, setof functions, ...).

Rendered schema, before / after

create type post_status as enum ('draft', 'published');
create table posts (id serial primary key, slug text not null unique, status post_status not null default 'draft');

Before (no enum, no sequence, constraint names dropped):

create table "posts" (
  "id" integer not null default nextval('posts_id_seq'::regclass),
  "slug" text not null,
  "status" post_status not null default 'draft'::post_status,
  PRIMARY KEY (id),
  UNIQUE (slug)
);

After (same style as Postgres drafts):

create type "public"."post_status" as enum ('draft', 'published');

create sequence "public"."posts_id_seq";

create table "public"."posts" (
    "id" integer not null default nextval('posts_id_seq'::regclass),
    "slug" text not null,
    "status" post_status not null default 'draft'::post_status
      );

alter sequence "public"."posts_id_seq" owned by "public"."posts"."id";

CREATE UNIQUE INDEX posts_pkey ON public.posts USING btree (id);

CREATE UNIQUE INDEX posts_slug_key ON public.posts USING btree (slug);

alter table "public"."posts" add constraint "posts_pkey" PRIMARY KEY using index "posts_pkey";

alter table "public"."posts" add constraint "posts_slug_key" UNIQUE using index "posts_slug_key";

This is what the UI's live-schema pane shows now.

Still broken

migra creates every function and composite type after the tables. So a table whose default, check or generated column calls a user function, or that has a composite-type column, fails, in drafts too. Two test.fails cases pin it. It's a migra draft-ordering change that reshuffles fixture output, so it's left for its own PR (listed in tasks/pg.md).

Also listed in the guide: custom ICU collations lose locale/deterministic; comments and grants are never drafted.

Change Purpose
packages/pg/src/impl/schema.ts: renderCanonicalSchema + its pg_catalog queries replaced by renderSchema (migra from PostgreSQL.empty() to the inspected db) One schema model for both rendering and diffing, so a rendering always diffs as identical to its source.
omitTables in the same file excludedTables (e.g. sqlfu_migrations) drops the table plus its indexes, constraints, triggers, policies and owned sequences from the inspector.
Leave out the public schema and plpgsql extension Every new database has them; create schema if not exists "public" was noise in the UI pane.
vendor/migra/migra.ts, changes.ts: create new domains before tables migra created domains after the tables using them (type "email" does not exist). Modified domains keep the old drop-then-create spot.
vendor/schemainspect/pg/obj.ts: create domain ends with ; Without it the domain merged into the next statement.
vendor/schemainspect/queries.ts: enums query uses typtype = 'e' typcategory = 'E' also matched domains over enums, so they were created twice.
vendor/schemainspect/inspected.ts: collation in ColumnInfo.creation_clause create table dropped collate "C"; only add column kept it.
packages/pg/test/schema-round-trip.test.ts The spec: each shape rendered, then diffed both ways. Also extractSchemaFromClient on a live db and excludedTables.
docs/guides/postgres.md, guides/pgkit.md, packages/sqlfu/README.md Known gaps shrink to what's still true; drop "use uuid or text ids for now" and the pgkit "schema coverage" caveat.
tasks/pg.md, tasks/complete/2026-10-29-pg-lossless-schema.md Step three item ticked; function/composite-type ordering added as the next item. Task file has the probe results and decisions.

Tested: @sqlfu/pg typecheck + full suite (postgres:16 from packages/pg/test/docker-compose.yml), and packages/ui/test/pg-studio.spec.ts (3/3).

Session: 7f20d0b1-4a1e-4bfc-9594-63bff2958ec5

🤖 Generated with Claude Code

Task file for replacing @sqlfu/pg's hand-rolled pg_catalog renderer
with the vendored migra, so materializeSchemaSql/extractSchemaFromClient
round-trip real Postgres schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pkg-pr-new

pkg-pr-new Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/sqlfu@161

commit: b556456

materializeSchemaSql/extractSchemaFromClient output, diffed against the
original schema in both directions, should be empty. Fails today for
serial, identity, enums, domains, generated columns, functions,
triggers, matviews, extensions and non-public schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mmkal

mmkal commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author

🤖 Superseded by #164 (this one closed automatically when its base branch pg-docs was deleted after #156 merged).

mmkal added a commit that referenced this pull request Oct 29, 2026
7c7e176 meant to fold tasks/complete/2026-10-29-pg-lossless-schema.md
into the shared task's "Schema renderer" section but only deleted it.
This restores that spec (condensed), ticks the section, points tasks/pg.md
at #159 instead of the closed #161, and moves the task to complete now
that all three sections are done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant