[jeandet]

Read CDF files straight from S3 (or any remote store)

On S3 you pay per byte and per request. This PR reads a remote CDF lazily: only the records you need, each fetched once.

How it works

1. You give pycdfpp.load a URL. It goes through fsspec, so s3://, gs://, az:// and https:// all work. For a source fsspec doesn't know, you can pass your own reader: any object with size() and read_ranges(ranges).
2. CDFpp fetches only the pages each record needs. It fetches them in batches, and keeps them in a per-process cache.
3. The cache sits in a reserved memory region. Pages stay unreadable until fetched, so a parser bug that reads outside its record crashes instead of returning garbage.

This runs as is against NASA's public HelioCloud bucket (pip install pycdfpp[s3]):

import pycdfpp

cdf = pycdfpp.load("s3://gov-nasa-hdrl-data1/spdf/cdaweb/data/ace/mag/level_2_cdaweb/"
                   "mfi_h0/2020/ac_h0_mfi_20200101_v07.cdf")
b_gse = cdf["BGSEc"].values  # fetches only this variable

With no AWS credentials on the machine, s3:// reads anonymously, so public buckets need no setup. Credentials that exist are always used. storage_options goes to fsspec and replaces this default.

Settings

Every setting has a documented default. You can override it per call or with a CDFPP_REMOTE_* environment variable, so a JupyterHub (HelioCloud) can preset them.

• read_ahead: 1 MiB by default. The reading guide explains why.
• tail_prefetch
• backing: memory, or a temp file
• temp_dir
• download_whole_below: opt-in only. Many small files read partially would cost more if each one were downloaded whole.

Measured on HelioCloud (public S3)

The ACE file above is 53.6 MB. Downloading it whole and loading it takes 4.2 s. One round trip costs about 0.5 s from my machine. With the default settings, through fsspec:

Workload	Time	Measured cost
Open the file	0.66 s	2.3 MB downloaded
Load one variable	1.15 s	8.6 MB downloaded
Load three variables	5.6 s	17 round trips
Load every variable	10.5 s	30 round trips

Remote reads win when you open a file or read a few variables. Reading most of the file is slower than downloading it, because each variable's index is a chain of records, read one link at a time.

fsspec and obstore give the same numbers. The cost comes from round trips, not from Python, so a C++ transport isn't needed for now.

Other changes

• GIL deadlock fixed. A thread holding the GIL could wait on a variable's lock while the loading thread needed the GIL to call the Python reader. Waiting on a variable's lock now releases the GIL.
• Decode threads never fetch. All fetching happens on the calling thread, before decoding starts.
• Corrupt or truncated data raises. A remote file never yields silent zeros or partial bytes.
• wait_for_variable_lock hook in variable.hpp.

Tests

• A new C++ suite, remote_loading. It checks that every test resource loads the same through the remote path as from disk, eagerly and lazily, page by page. It also checks batching, corrupt offsets, truncated records, and the default window.
• New Python tests for reader errors and retry, bad chunks (short, missing, strided), settings and environment overrides, the deadlock, and exiting with a live lazy CDF.
• The whole python_loading suite also runs a second time through a reader.
• 48/48 pass locally.

Before merging

• subprojects/cpp_utils.wrap points at cpp_utils main (29e4469), which now holds the remote-read code.
• Catch2 got a patch bump (3.16.1) from meson wrap update.

Left for later

• Partial reads of a single record.
• WASM. The remote path is compiled out under Emscripten.
• Free-threaded Python.
• A C++ transport, if Python ever becomes the bottleneck.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 97.15100% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.22%. Comparing base (6d45a47) to head (b210f57).

Files with missing lines	Patch %	Lines
pycdfpp/remote.hpp	93.84%	4 Missing ⚠️
include/cdfpp/cdf-io/loading/buffers.hpp	86.66%	2 Missing ⚠️
pycdfpp/_remote.py	92.00%	2 Missing ⚠️
tests/remote_loading/main.cpp	98.78%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #135      +/-   ##
==========================================
- Coverage   96.24%   96.22%   -0.03%     
==========================================
  Files          91       94       +3     
  Lines        8824     9201     +377     
==========================================
+ Hits         8493     8854     +361     
- Misses        331      347      +16     

Flag	Coverage Δ
unittests	96.22% <97.15%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
35 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud