schemachange: pair indexes across a retyped column (ST-5, D8) - #138
Conversation
48d61e1 to
b54bd32
Compare
ALTER COLUMN ... TYPE re-creates every index on that column for the new type, so pg_index.indclass on the shadow's copy flips to the new type's default operator class (int4_ops -> int8_ops) while the index still covers the same columns the same way. The gate's exact-definition pairing left such indexes unpaired, and a swap would have kept the LIKE-derived name (_pgsprite_<hash>_new_pkey) on the live table. GateCutover now derives the set of columns the statement retyped (same name on both sides, different canonical type) and runs a second pairing pass over the exact pass's leftovers only, with the operator class and collation of every retyped key column set aside. Access method, uniqueness, columns, ordering, predicate, and backing constraint must still agree; an expression over a retyped column and every untouched column stay exact. Pairs keep the source order. Unit tests cover the retyped-column derivation (including a quoted name), the relaxed pass over leftovers, and that only opclass/collation are relaxed. Integration tests gate an int -> bigint change on a table with a PK, a DESC index on the retyped column, and an index on an untouched column, asserting the int4_ops/int8_ops precondition and full pairing; and that a dropped column's index still stays unpaired alongside a retyped one. Design D8 and invariant ST-5 record the one set-aside difference.
de2c3a0 to
5d6f315
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
🤖 1/2: adversarial correctness review of 1 blocking, 3 non-blocking. The two-pass shape is right. The exact pass runs first, the relaxed pass sees only its leftovers, and an index on a column the statement did not retype pairs exactly or not at all. The gate stays fail-closed: every refusal before the pairing is unchanged, and extra pairs only widen the constraint-name check. The blocking finding is in what the relaxed pass sets aside. It blanks every opclass and collation on a retyped key column, including the ones the user wrote explicitly. The server keeps those on a rebuild, so they still tell indexes apart. Blocking1. The relaxed pass blanks explicit collations, so two indexes that differ only in collation pair by name order and swap names (ST-5, D8). D8 relies on one premise: two indexes with the same key are interchangeable, so any pairing between them restores an equivalent catalog. The relaxed key breaks it. Here is the example I ran. The gate mints The fix is to relax only what the server re-derives. Test that fails on 5d6f315 and passes with the fix// Two indexes on one column that differ only in an explicit collation stay
// distinct across a retype: the server keeps the explicit collation and
// re-derives only the implicit one, so each source index pairs with the
// shadow index that carries its own collation (ST-5, D8).
func TestGateCutoverRetypeKeepsAnExplicitCollationApart(t *testing.T) {
f := newShadowFixture(t)
f.createAccounts(t)
f.exec(t, `CREATE INDEX accounts_label_plain ON %s.accounts (label)`)
f.exec(t, `CREATE INDEX accounts_label_c ON %s.accounts (label COLLATE "C")`)
s := f.stage(t, "accounts", `ALTER TABLE %s.accounts ALTER COLUMN label TYPE char(20)`)
shadow := s.built.ShadowTable()
ready, err := f.gate(t, s)
require.NoError(t, err)
pairs := ready.Indexes().Pairs
assert.Contains(t, pairs, schemachange.DependentPair{Kind: schemachange.DependentIndex, SourceName: "accounts_label_plain", ShadowName: shadow + "_label_idx"})
assert.Contains(t, pairs, schemachange.DependentPair{Kind: schemachange.DependentIndex, SourceName: "accounts_label_c", ShadowName: shadow + "_label_idx1"})
}
Non-blocking1. The collation half of the relaxation is untested. Mutant M7 keeps every collation, and every test still passes. The PR's integration tests retype Test that passes on 5d6f315 and kills M7// Retyping an integer column to text gives its index a collation as well as
// a new operator class; both are set aside, so the index still pairs (D8).
func TestGateCutoverPairsAnIndexAcrossARetypeThatAddsACollation(t *testing.T) {
f := newShadowFixture(t)
f.createAccounts(t)
s := f.stage(t, "accounts", `ALTER TABLE %s.accounts ALTER COLUMN balance TYPE text`)
shadow := s.built.ShadowTable()
ready, err := f.gate(t, s)
require.NoError(t, err)
assert.Contains(t, ready.Indexes().Pairs, schemachange.DependentPair{Kind: schemachange.DependentIndex, SourceName: "accounts_balance_idx", ShadowName: shadow + "_balance_idx"})
}It passes on 2. The exact pass pairs the only shadow index, so Test that passes on 5d6f315 and kills M11 and M17// The relaxed pass sees only what the exact pass left on both sides: with a
// leftover on each side it runs, and still cannot hand the shadow index the
// exact pass already took to a second source index.
func TestPairIndexesRelaxedPassRunsOnlyOverExactLeftovers(t *testing.T) {
source := []indexEntry{
{name: "orders_id_idx", definition: btreeOn("pg_catalog.int4_ops", "id")},
{name: "orders_id_idx1", definition: btreeOn("pg_catalog.int8_ops", "id")},
}
shadow := []indexEntry{
{name: "_new_id_idx", definition: btreeOn("pg_catalog.int8_ops", "id")},
{name: "_new_qty_idx", definition: btreeOn("pg_catalog.int4_ops", "qty")},
}
got, err := pairIndexes(source, shadow, map[string]bool{"id": true})
require.NoError(t, err)
assert.Equal(t, DependentPairing{
Pairs: []DependentPair{{Kind: DependentIndex, SourceName: "orders_id_idx1", ShadowName: "_new_id_idx"}},
UnpairedSource: []string{"orders_id_idx"},
UnpairedShadow: []string{"_new_qty_idx"},
}, got)
}It passes on 3. A predicate over a retyped column stays unpaired, and the gate still passes. The design text names only expressions as held to an exact match, but the server re-renders predicates too. Verified
This review was generated by Claude Code (claude-opus-5-5). |
|
🤖 2/2: OSS adoption and integration ease, at This closes the most visible gap #137 left. A widened key is the most common type change, and keeping 1. Say which pairs the relaxed pass made. An operator reviewing a cutover preview should see the difference between "this index is unchanged" and "this index is re-created for 2. Give the remaining unpaired cases a reason. This narrows the unpaired set, so what is left is more likely to surprise someone. That covers an expression or a predicate over a retyped column (1/2, non-blocking 3), and a column the statement dropped. The suggestion from #137 still applies here: a reason on each unpaired entry would let an importer say "this index goes away" only when the statement dropped its column. Today the cases look the same as a pairing the engine could not make. 3. Document how a build fails when the new type rejects an index. A retype whose index opclass does not accept the new type, such as BRIN This review was generated by Claude Code (claude-opus-5-5). |
aparajon
left a comment
There was a problem hiding this comment.
🤖 Approving 5d6f315 with 1 blocking finding: the relaxed pass blanks explicit collations as well as derived ones. So two indexes on a retyped column that differ only in COLLATE pair by name order, and each ends up with the other's name (ST-5, D8). The fix is to set aside only the default opclass and the column's own collation. The 1/2 comment has the blocking finding with its test and a fix that keeps the package green, plus three non-blocking findings. Two of them come with tests: the collation half of the relaxation is unpinned, and the leftovers-only test never reaches the relaxed pass. The 2/2 comment has three non-blocking integration notes.
This stamp was left by Claude Code (claude-opus-5-5).
morgo
left a comment
There was a problem hiding this comment.
🤖 Automated adversarial review, posted on Morgan Tocker's behalf.
Approving. Running the relaxation as a second pass over only the first pass's leftovers is the right structure — it keeps the exact match authoritative, so an index that already has a perfect counterpart can never be stolen by a looser one, and it bounds the blast radius of the relaxation to the indexes that actually failed. Setting aside the operator class and collation rather than the whole key column is also the narrow version of the fix: everything that decides which rows the index covers still has to agree.
Verified rather than assumed:
- The merge cannot drop an index out of both unpaired lists.
merged.UnpairedSource/UnpairedShadoware taken wholesale from the relaxed pass, which looks like it would lose the exact pass's leftovers — but the relaxed pass is fed exactlyexact.UnpairedSource/UnpairedShadow, so every exact leftover ends up either inpartneror in the relaxed pass's own unpaired list. Worth checking becausesourceNames()isPairsplusUnpairedSource, and that is whatconfirmNamesFreechecks the old-name space against; a name silently dropped here would be a derived name the gate never confirms is free, and the swap would fail under the cutover lock instead of refusing before it. - The early return is consistent with the merge.
len(exact.UnpairedShadow) == 0returnsexactwhole, which carries its ownUnpairedSource; the merge path only replaces those lists when the relaxed pass actually ran over them. relaxedAcross's shallow copy does not reach back into the caller's entries.relaxed := dshares the backing arrays ofKeyColumns,IncludedandOptions, but only the freshly clonedOpclassesandCollationsare written. SincepairIndexesrenders the same[]indexEntryslice twice, an in-place clear would have rewritten the inputs between passes.- A constraint-backed index cannot cross-pair with a plain one.
Constraintstays in the relaxed key, soaccounts_pkeyin the new integration test pairs through its own constraint definition rather than competing withaccounts_balance_idx, even though both key on the retyped column. - The bare-and-sanitized keying matches the catalog form.
pg_get_indexdef(…, k, true)prints the column name quoted only where quoting is required, andpgx.Identifier{}.Sanitize()always quotes, so entering both covers a case-sensitive name without a separate unquoting step.
Worth fixing before merge
The relaxed pass pairs by a key that is no longer unique, and pairByDefinition's "identical definitions are interchangeable" justification does not survive that. pairByDefinition takes candidates[0] when several dependents share a key. That is sound under exact matching — the doc's reasoning is that two source indexes with the same definition really are the same index twice, so an arbitrary pairing restores an equivalent catalog. After relaxedAcross the key is deliberately lossy, and two indexes that differ only in the thing it erased now collide.
Concretely, on a column the statement retypes:
CREATE INDEX accounts_code_idx ON accounts (code);
CREATE INDEX accounts_code_pattern_idx ON accounts (code bpchar_pattern_ops);
ALTER TABLE accounts ALTER COLUMN code TYPE text;Both source indexes relax to the same key, both shadow indexes relax to the same key, and the pairing between them is positional. If the two lists do not happen to agree in order — the source is ordered by ic.relname, the shadow by whatever names LIKE-built index creation produced — the swap puts accounts_code_idx on the pattern-ops index and accounts_code_pattern_idx on the default one. Every check the gate makes still passes, because the set of names is unchanged; the catalog afterwards has two of the operator's own indexes wearing each other's names. The same shape arises from the collation half (CREATE INDEX … (name) alongside CREATE INDEX … (name COLLATE "C")), which is if anything the more common pair, since both are the standard ways to make LIKE indexable.
TestPairIndexesRelaxedPassUsesOnlyExactLeftovers covers the adjacent hazard — a relaxed candidate stealing an index the exact pass already spoke for — but not this one, where both candidates are genuinely leftovers and the ambiguity is created by the relaxation itself. The cheap fix is to refuse to relax an ambiguous key: if more than one source leftover or more than one shadow leftover shares a relaxed key, leave all of them unpaired rather than guessing. That keeps the current outcome for every case the PR is aimed at and degrades to today's behaviour for the ambiguous ones, which is the conservative direction for a fidelity gate.
Notes
An index whose key columns are untouched can still be knocked out of pairing by a retype, and nothing in the relaxation can reach it. Predicate is pg_get_expr(indpred, …), and the deparser prints a constant with an explicit cast for every type other than the handful it treats as self-evident — int4 among them, int8 not. So CREATE INDEX ON accounts (created_at) WHERE balance > 0, with balance retyped int4 → int8, reads (balance > 0) on the source and (balance > '0'::bigint) on the shadow. The key column is created_at, which the statement did not touch, so relaxedAcross clears nothing and the index stays unpaired and keeps its generated shadow name after cutover.
The doc change sets aside the expression case explicitly — "an expression over such a column … is held to an exact match" — and I read that as deliberate. The predicate case is covered by the same sentence only implicitly ("everything else about the two definitions must still agree"), and it is the one that will surprise, because the index that loses its name does not look like an index on a retyped column at all. Either name predicates alongside expressions in that sentence, or say in pairIndexes' comment which parts of the definition the relaxation can and cannot reach.
The relaxation is keyed on schemadiff.Model type strings, which makes the gate's behaviour depend on how canonical those are. retypedColumns flags a column when targetType != c.Type as plain strings. A spelling difference that is not a type difference — int8 against bigint, varchar(64) against character varying(64) — relaxes a column that was never retyped, and relaxing a column that did not change can only loosen the gate: two indexes on that column differing genuinely in operator class would then pair. That is the same ambiguity as above arriving by a different route, so if the model's types are already canonicalised, a one-line assertion of that in retypedColumns' comment is worth more than it costs; if they are not, the ambiguity guard above handles both.
indexesNamed is quadratic, which is fine here and worth a word. slices.Contains over the leftover names, once per index, is nothing at catalog index counts — but it reads like an oversight next to the map the rest of the file uses. A one-line note that the leftover lists are small by construction (they are the residue of an exact pass) would stop the next reader from "fixing" it.
|
🤖 Adversarial review response — created by Kiran's code review agent (Amp, Claude Opus 4.6) — pull/138, follow-up commit The relaxed pass now sets aside only what the server re-derives on a retyped key column (default operator class, column's own collation), a relaxed definition shared by more than one index on either side pairs none of them, and the docs name predicates, canonical types, and build-time server errors. The blocking finding and the "worth fixing before merge" item are fixed; the two integration-ease suggestions that need a new field on the pairing are tracked as internal follow-ups.
Verified sections of both reviews (merge cannot drop an index from both unpaired lists, early return consistent with the merge, shallow copy does not reach the caller's entries, constraint-backed indexes cannot cross-pair, bare-and-sanitized keying) need no action; the merge path is unchanged and the new ambiguity exclusion feeds the same Source: block/pg-sprite#138, review comments 5931555831 and 5931557266, reviews 5379260375 and 5383581722 at head |
The relaxed pairing rule for a key column the gated statement retyped set aside the operator class and collation outright, so an index with a written operator class or COLLATE clause paired with a shadow index that lost it. Narrow the rule to what the server re-derives: the default operator class (pg_opclass.opcdefault) and the column's own collation (indcollation = attcollation). A written opclass or collation still has to agree. Leave a relaxed definition that two indexes on either side share unpaired rather than pairing by position: identical indexes are interchangeable only when their full definitions match, and the relaxation cannot show that. Document that expressions and predicates over a retyped column stay exact, that schemadiff types are canonical (format_type), and that a server error from applying the statement to the empty shadow is a wrapped *pgconn.PgError routed by SQLSTATE, not a RefusalCause. Tests: explicit opclass and explicit collation kept apart across a retype (unit and PG), an added collation still pairs, the relaxed pass runs only over exact leftovers, and shared relaxed keys on either side stay unpaired.
Closes the known gap carved out of #137: the cutover gate's index pairing now follows a column across a type change.
Why
ALTER COLUMN id TYPE bigintre-creates every index onidfor the new type. On PostgreSQL 14–18 the shadow's copy of the primary key and of any secondary index onidcarriesint8_opsinpg_index.indclasswhile the source's carriesint4_ops, even thoughpg_get_constraintdefstill printsPRIMARY KEY (id). #137 pairs indexes by exact catalog definition, operator classes included, so those indexes landed inUnpairedSource/UnpairedShadowand the swap would have left_pgsprite_<hash>_new_pkeyas the live table's primary-key name.What
pkg/schemachange/dependents_retype.go(new):retypedColumns(source, target)— the columns both models carry under one name with a different canonical type. Recorded both bare andpgx.Identifier{...}.Sanitize()-quoted, sincepg_get_indexdefquotes a key column only when it needs to.indexDefinition.relaxedAcross(retyped)— sets aside only what the server re-derives on a retyped key column: the type's default operator class (pg_opclass.opcdefault) and the column's own collation (indcollation = attcollation). An operator class orCOLLATEwritten in the index still has to agree. Expressions and predicates are never relaxed (the server may render either differently for the new type).pairIndexes(source, shadow, retyped)— exact pass first, then a relaxed pass over the exact pass's leftovers only, merged in source order. A relaxed definition that more than one index on either side renders pairs none of them — the relaxation cannot show those indexes are interchangeable, so they stay unpaired rather than pairing by position. An index on an untouched column pairs exactly or not at all; a predicate,DESC/NULLSordering, uniqueness, or access-method difference still leaves the pair unpaired.readIndexesreads the two per-key facts the relaxation needs (opcdefault, and whether the key's collation is the column's own); they ride onindexDefinitionoutside the pairing key.GateCutoverpassesretypedColumns(sourceModel, targetModel)into the pairing;indexDependentsrenders the relaxed definition so the pairing key and the unpaired report agree.docs/copy-and-swap-design.mdand ST-5 indocs/invariants.mdrecord the one set-aside difference and the ambiguity rule; thepkg/schemachangepackage-map row says the same.docs/refusal-classes.mdnotes that a server error from applying the statement to the empty shadow (e.g. an operator class that does not accept the new type, SQLSTATE 42804) is a wrapped*pgconn.PgErrorrouted by SQLSTATE, not aRefusalCause.Tests
dependents_retype_test.go): retyped-column derivation incl. a mixed-case"Qty"; pairing across a retyped column keeps source order; an opclass difference on an untouched column stays unpaired; only the default opclass and own collation are relaxed (a predicate orindoptiondifference still fails; an explicit opclass or explicit collation is kept across the retype); an expression over a retyped column is not relaxed; the relaxed pass runs only over exact leftovers; a relaxed key shared by two source or two shadow indexes leaves all of them unpaired.cutover_gate_retype_integration_test.go, PG16 testcontainers):TestGateCutoverPairsIndexesAcrossARetypedColumn—accounts(id integer PRIMARY KEY, balance integer, label text)withaccounts_id_desc_idx (id DESC)andaccounts_balance_idx; gatesALTER TABLE accounts ALTER COLUMN id TYPE bigint; asserts theint4_opsvsint8_opsprecondition on the two primary keys and that all three indexes pair with nothing left over.TestGateCutoverRetypePassLeavesADroppedColumnsIndexUnpaired—ALTER COLUMN id TYPE bigint, DROP COLUMN balance; theidindexes pair,accounts_balance_idxstays inUnpairedSource.TestGateCutoverPairsAnIndexAcrossARetypeThatAddsACollation—ALTER COLUMN balance TYPE text; the index gains a collation (""→default) andtext_ops, and still pairs.TestGateCutoverRetypeKeepsAnExplicitCollationApart—accounts_label_plain (label)andaccounts_label_c (label COLLATE "C"), retypelabeltochar(20); each pairs with the shadow index carrying its own collation.TestGateCutoverRetypeKeepsAnExplicitOpclassApart—(label)and(label bpchar_pattern_ops)on achar(20)column, retype totext COLLATE "C"; the explicit opclass is kept and the two indexes pair with their own counterparts.SKIP_INTEGRATION=1 go test ./...,go test -race ./pkg/schemachange/,make lint(0 issues) all pass locally.Stack
Based on #137 (
kiran01bm/cs9-fidelity-gate), which is based on #136 (kiran01bm/cs6-repair-policy). Retarget to the parent's base as each merges. No capability, verdict, or CLI surface changes;demo/tour.shis unaffected.