Skip to content

Flink: Add periodic and timed reload of the lookup join full cache - #18149

Open
Guosmilesmile wants to merge 2 commits into
apache:mainfrom
Guosmilesmile:lookup_join_reload
Open

Guosmilesmile wants to merge 2 commits into
apache:mainfrom
Guosmilesmile:lookup_join_reload

Conversation

@Guosmilesmile

Copy link
Copy Markdown
Contributor

Follow up #18144. Part of #18142

Add support for reloading the full cache of an Iceberg lookup join, instead of keeping the initial snapshot for the lifetime of the job.

The reload is opt-in, so the existing behavior remains unchanged unless a reload option is configured.

Changes

  • Support the standard Flink full cache reload options:
  • lookup.full-cache.reload-strategy (PERIODIC or TIMED)
  • lookup.full-cache.periodic-reload.interval / .schedule-mode
  • lookup.full-cache.timed-reload.iso-time / .interval-in-days
  • Infer the reload strategy from the configured options when it is not explicitly set.
  • Use the standard Flink reload triggers.
  • Reload from the current snapshot and atomically replace the existing cache after the new cache is fully loaded.
  • Skip the reload when there is no new snapshot.
  • Fail the job on the next lookup if a reload fails, instead of continuing to serve the previous cache. Since Flink triggers swallow reload failures, the lookup function maintains and latches the failure state.
  • Keep the initial cache load separate from reloads. A trigger only reloads an already-loaded cache, avoiding concurrent access to the same Table instance from the initial lookup and the reload thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant