Skip to content

Core: Add V4 scan task planner - #17541

Open
anoopj wants to merge 20 commits into
apache:mainfrom
anoopj:v4-native-planner
Open

anoopj wants to merge 20 commits into
apache:mainfrom
anoopj:v4-native-planner

Conversation

@anoopj

@anoopj anoopj commented Aug 6, 2026

Copy link
Copy Markdown
Member

Adds the planner that walks a v4 adaptive metadata tree and emits FileScanTasks. It reads the root manifest, emits a task per live DATA entry, and expands DATA_MANIFEST entries into their leaf manifests. Leaves are read lazily and, when a worker pool is supplied, expanded in parallel.

Keeping it package-private and not wired into any scan yet.

Support for the following features will be added in followup PRs.

  • Inherited tracking: sequence numbers, snapshot id, and first-row-id aren't applied
  • Manifest-level pruning: e.g. using the manifest_info count based short-circuit, stats/partition pruning
  • Delete content, manifest deletion vectors, encryption, and non-v4 leaves
  • Content cache + eager-fetch of manifests

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

7 participants