Skip to content

Use stable PG19 Beta 4 images and refresh reindexed metadata - #8881

Merged
ibrahim halatci (ihalatci) merged 4 commits into
mainfrom
ihalatci-pg19-beta4-stable-images
Sep 28, 2026
Merged

ibrahim halatci (ihalatci) merged 4 commits into
mainfrom
ihalatci-pg19-beta4-stable-images

Conversation

@ihalatci

@ihalatci ibrahim halatci (ihalatci) commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • replace the PostgreSQL 19 Beta 4 development image suffix with the published immutable suffix -vc231ab7
  • invalidate cached metadata index OIDs whenever concurrent reindex replaces one of the 23 indexes stored in MetadataCache
  • preserve the worker-node cache in WAL senders so logical decoding never rebuilds it by scanning pg_dist_node
  • fix the repeatable PG19 vanilla create_view failure and the resulting shard-split isolation hang

Root cause

PostgreSQL's vanilla create_index test runs REINDEX SCHEMA CONCURRENTLY pg_catalog in parallel with create_view. Concurrent reindex replaces Citus catalog indexes incrementally rather than atomically.

The initial implementation reset metadata OIDs only when the pg_dist_object table was invalidated. That fixed stale pg_dist_object_pkey OID 16757, but a parallel backend could repopulate a later cached index while reindex was still progressing. CI subsequently failed on stale pg_dist_schema_pkey OID 17321.

The final implementation recognizes relcache invalidations for every index OID stored in MetadataCache and resets the cached OIDs each time one is replaced. The list covers all 23 *IndexId fields exactly, without catalog lookups inside the relcache callback or resets for unrelated relations.

A full metadata reset is unsafe in WAL senders: it invalidates the worker-node cache, which then attempts to scan pg_dist_node inside logical decoding. PostgreSQL rejects that non-catalog scan, shard-split replication workers repeatedly restart, and create_distributed_table_concurrently waits indefinitely for replication catch-up. WAL senders therefore clear only MetadataCache, while ordinary backends retain the full metadata-system reset.

Stable image evidence

Validation

  • PG19 non-cassert vanilla: 239/239 passed in 10 consecutive runs
  • PG19 cassert vanilla: 239/239 passed, including create_view
  • PG19 cassert isolation_create_distributed_table_concurrently: passed six final-candidate runs in 17-20 seconds
  • PG19 cassert multi_1_create_citus_schedule: 38/38 passed, including multi_replicate_reference_table and pg19
  • PG17.11 and PG18.6 builds: passed without source compiler warnings
  • PG19 Beta 4 cassert build/install with CI flags: passed
  • explicit cache audit: all 23 MetadataCacheData *IndexId fields covered exactly once
  • full repository style workflow and git diff --check: passed

Follow-up to #8870 and citusdata/the-process#250.

ihalatci-msft and others added 2 commits September 27, 2026 13:49
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.76%. Comparing base (0cd3fe3) to head (83c3ef7).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8881      +/-   ##
==========================================
- Coverage   88.76%   88.76%   -0.01%     
==========================================
  Files         290      290              
  Lines       65102    65132      +30     
  Branches     8220     8223       +3     
==========================================
+ Hits        57790    57816      +26     
- Misses       4937     4941       +4     
  Partials     2375     2375              
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ihalatci-msft and others added 2 commits September 27, 2026 21:42
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@ihalatci
ibrahim halatci (ihalatci) merged commit b4faf35 into main Sep 28, 2026
132 checks passed
@ihalatci
ibrahim halatci (ihalatci) deleted the ihalatci-pg19-beta4-stable-images branch September 28, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants