Skip to content

Expose pooled CUDA file scans through FFI#8788

Merged
onursatici merged 2 commits into
developfrom
os/pinned-ffi
Jul 16, 2026
Merged

Expose pooled CUDA file scans through FFI#8788
onursatici merged 2 commits into
developfrom
os/pinned-ffi

Conversation

@onursatici

Copy link
Copy Markdown
Contributor

Rationale for this change

We don't expose a way in ffi to use the pinned buffer pool on reading vortex files into the GPU

What changes are included in this PR?

cuda session now owns the buffer pool, and that is shared on scans done with this context

@onursatici onursatici added the changelog/feature A new feature label Jul 16, 2026
@onursatici onursatici requested a review from 0ax1 July 16, 2026 13:17
Signed-off-by: Onur Satici <onur@spiraldb.com>
Signed-off-by: Onur Satici <onur@spiraldb.com>
@onursatici onursatici enabled auto-merge (squash) July 16, 2026 13:25
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done 61345d8 1 Explore Profiling Data
Previous Runs (1)
Status Commit Job Attempt Link
🟢 Done eb5b635 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Vortex queries

Verdict: No clear signal (low confidence)
Attributed Vortex impact: -0.6%
Engines: DataFusion No clear signal (+9.1%, low confidence) · DuckDB Likely improvement (-9.5%, medium confidence)
Vortex (geomean): 0.840x ✅
Parquet (geomean): 0.843x ✅
Shifts: Parquet (control) -15.7% · Median polish -16.2%

How to read Verdict and Engines
  • Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
  • Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
  • Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (0.857x ✅, 2↑ 0↓)
name PR 61345d8 (ns) base e727c46 (ns) ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 🚀 10052101 11634784 0.86
vortex_q01/datafusion:vortex-file-compressed 🚀 6556625 7718973 0.85
datafusion / parquet (0.785x ✅, 2↑ 0↓)
name PR 61345d8 (ns) base e727c46 (ns) ratio (PR/base)
vortex_q00/datafusion:parquet 🚀 20602447 25949078 0.79
vortex_q01/datafusion:parquet 🚀 4727027 6089901 0.78
duckdb / vortex-file-compressed (0.819x ✅, 2↑ 0↓)
name PR 61345d8 (ns) base e727c46 (ns) ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed 🚀 10053260 12065527 0.83
vortex_q01/duckdb:vortex-file-compressed 🚀 5954243 7392324 0.81
duckdb / parquet (0.905x ➖, 1↑ 0↓)
name PR 61345d8 (ns) base e727c46 (ns) ratio (PR/base)
vortex_q00/duckdb:parquet 🚀 23546974 26698649 0.88
vortex_q01/duckdb:parquet 9362288 10071331 0.93

No file size changes detected.

@onursatici onursatici merged commit 01e5eb0 into develop Jul 16, 2026
77 checks passed
@onursatici onursatici deleted the os/pinned-ffi branch July 16, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants