feat: split raw data storage into per-dataset SQLite files

[astafan8]

Summary

This PR implements split raw data storage for QCoDeS: an opt-in feature that writes raw measurement data (results table rows) into individual per-dataset SQLite files while keeping all metadata in the main database. The goal is to prevent the main DB file from growing excessively large as datasets accumulate, so that a user can keep a single lightweight metadata database on their machine instead of many ever-growing ones.

It also includes dataset management helpers for maintaining the main database and raw data files over time.

Motivation

The main QCoDeS SQLite database stores both metadata (experiments, runs, parameter layouts, dependencies) and raw measurement data (results tables) in a single file. Over time this file can grow to many gigabytes, slowing down operations that only need metadata. By splitting the raw data into per-dataset files, the main DB stays lightweight while data integrity is preserved.

Design

Transparent routing via _data_conn

• A single _data_conn property on DataSet is the routing point for all results-data read/write operations. It returns the per-dataset raw data connection when split storage is enabled, otherwise the main DB connection.
• All write paths (add_results, _BackgroundWriter) and read paths (get_parameter_data, DataSetCacheWithDBBackend, number_of_results, __len__) go through it.
• Zero changes to the public DataSet API — every existing method behaves identically whether the feature is on or off.

Config — follows the existing export-path pattern

Two new options in the dataset section of qcodesrc.json:

• raw_data_to_separate_db (bool, default false) — enables split storage.
• raw_data_path (string, default "{db_location}") — folder for per-dataset files. {db_location} expands to a folder derived from the main DB path (reusing _expand_export_path).

Per-dataset files — lightweight, GUID-named

• Each file is named <guid>.db and contains only the results table + numpy type adapters — no QCoDeS metadata schema.
• Created via the shared _create_run_table() / _connect_to_sqlite_file() helpers so the schema and connection settings stay identical to the main DB.

No empty results table in the main DB

• When the feature is enabled, no results table is created in the main database at all — it stores only run metadata. This mirrors how DataSetInMem (the netcdf-backed dataset) records runs (create_run_table=False).
• Split-storage runs are identified by a dedicated raw_data_db_path column in the runs table, recorded at dataset creation. This positively distinguishes them from DataSetInMem runs (which also have no results table) in _get_datasetprotocol_from_guid, and is future-proof for non-SQLite raw-data backends.
• raw_data_db_path is an internal storage detail: it is stored in the runs table but excluded from the user-facing metadata dict, and read/written through get_raw_data_db_path_for_run / set_raw_data_db_path_for_run. This keeps it out of metadata comparisons (the_same_dataset_as) and out of extracted/exported databases.

Features that depend on the results table — handled explicitly

• Counting (number_of_results / __len__) return 0 when the results table does not exist (e.g. a pristine, not-yet-started split dataset).
• Subscribers: a subscription requested before the dataset is started (before the raw table exists) is deferred and materialised at start time. Factored into _queue_pending_subscriber / _create_and_start_subscriber / _start_pending_subscribers.
• Subscriber triggers, unsubscribe, unsubscribe_all operate on _data_conn so they target the DB that actually holds the results table.

Extract / export compatibility

• extract_runs_into_db reads results via _data_conn (from the per-dataset file), and the copied run does not carry over the source's raw_data_db_path.
• export_datasets_and_create_metadata_db works: the NetCDF path goes through get_parameter_data (already routed), and the copy-as-is fallback uses the extract fix above.

Shared connection helpers (deduplication)

• Extracted _connect_to_sqlite_file() and _register_numpy_sqlite_adapters_and_converters() in sqlite/database.py; both connect() and connect_to_raw_data_db() reuse them, so the numpy adapters and connection settings are defined once.

Dataset management helpers

Exposed via qcodes.dataset:

• update_raw_data_paths(db_path, new_raw_data_folder) — update stored paths after per-dataset files have been moved to a new folder.
• purge_orphaned_datasets(db_path, *, dry_run=True) — remove dataset records from the main DB whose raw data files no longer exist on disk.
• cleanup_datasets(db_path, *, older_than_days=None, sample_name=None, larger_than_mb=None, dry_run=True) — remove datasets (DB records AND raw data files) matching criteria (AND logic).

Both purge and cleanup default to dry_run=True and return dataclass results with full details. Dataset removal reuses the existing remove_dataset_from_db / _validate_table_name machinery.

Files changed

File	Type	Description
src/qcodes/dataset/_raw_data_storage.py	New	Storage config helpers, create_raw_data_db(), connect_to_raw_data_db(), update_raw_data_paths(), purge_orphaned_datasets(), cleanup_datasets(), result dataclasses
src/qcodes/dataset/data_set.py	Modified	_data_conn/_results_table_exists properties, _raw_data_db_path, routing in __init__, _perform_start_actions, add_results, get_parameter_data, number_of_results, __len__, _BackgroundWriter, deferred subscribers, unsubscribe/unsubscribe_all, split-vs-inmem disambiguation
src/qcodes/dataset/sqlite/queries.py	Modified	get_raw_data_db_path_for_run/set_raw_data_db_path_for_run, RawDataDatasetRecord dataclass + get_datasets_with_raw_data_path, remove_dataset_from_db, RAW_DATA_DB_PATH_COLUMN excluded from metadata
src/qcodes/dataset/sqlite/database.py	Modified	Shared _connect_to_sqlite_file() + _register_numpy_sqlite_adapters_and_converters() reused by connect()
src/qcodes/dataset/database_extract_runs.py	Modified	_extract_single_dataset_into_db reads via _data_conn
src/qcodes/dataset/data_set_cache.py	Modified	load_data_from_db() uses _data_conn
src/qcodes/dataset/subscriber.py	Modified	Trigger creation uses _data_conn
src/qcodes/dataset/__init__.py	Modified	Exports the new public helpers
src/qcodes/configuration/qcodesrc.json + qcodesrc_schema.json	Modified	New config defaults + schema
docs/dataset/introduction.rst, docs/dataset/dataset_design.rst, docs/examples/DataSet/Database.ipynb	Modified	Feature + management docs, design notes
docs/changes/newsfragments/8219.new	New	Towncrier news fragment
tests/dataset/test_raw_data_storage.py	New	Unit + integration + management + extract/export + no-empty-table tests

Verification

• tests/dataset/test_raw_data_storage.py: all pass (unit, integration, management, extract/export, subscribe-before-start, count-before-start, no-main-table).
• Full tests/dataset suite in default mode: no regressions.
• Validated with split storage globally enabled (temporary local harness, not part of the PR): remaining failures are all expected/inherent (tests that assert disabled-mode behaviour, low-level tests that call queries.get_parameter_data(ds.conn, ...) directly bypassing the DataSet API, and db-overview/guid-scan tests that inherently assume data in the main DB).
• Pyright: 0 errors. Ruff check + format: pass. pre-commit: pass.

[codecov]

Codecov Report

❌ Patch coverage is 93.31307% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.31%. Comparing base (8e5ea8a) to head (94884fc).
⚠️ Report is 75 commits behind head on main.

Files with missing lines	Patch %	Lines
src/qcodes/dataset/_raw_data_storage.py	94.68%	10 Missing ⚠️
src/qcodes/dataset/data_set.py	87.95%	10 Missing ⚠️
src/qcodes/dataset/sqlite/queries.py	95.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8219      +/-   ##
==========================================
+ Coverage   71.11%   71.31%   +0.20%     
==========================================
  Files         305      306       +1     
  Lines       32004    32308     +304     
==========================================
+ Hits        22760    23042     +282     
- Misses       9244     9266      +22     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[astafan8]

Added: update_raw_data_paths helper

Added a utility function for users who move their per-dataset raw data files to a new location. This mirrors the pattern used for exported netCDF files.

Usage:

`python
from qcodes.dataset import update_raw_data_paths

update_raw_data_paths(
db_path="/path/to/main_database.db",
new_raw_data_folder="/new/location/of/raw_files/"
)
`

The function:

• Scans all datasets in the main DB that have raw_data_db_path metadata
• For each, checks if the corresponding .db file exists in the new folder
• Updates the stored path to point to the new location
• Logs warnings for any files not found in the new folder

4 tests added, documentation updated in introduction.rst and Database.ipynb.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.

[astafan8]

ok, so instead of copying this over here from connect funcition, can't you extract that part into a function and re-use it in here and in connect? the same is true for the rest of this function

[astafan8]

Addressed in ae3c879. Extracted the shared logic into connect_to_sqlite_file() and _register_numpy_sqlite_adapters_and_converters() in sqlite/database.py, and now both connect() and connect_to_raw_data_db() reuse them. I also reused the existing _create_run_table() in create_raw_data_db() (instead of duplicating the results-table CREATE TABLE logic), and reused _validate_table_name() before the DROP TABLE in remove_dataset_from_db().

[jenshnielsen]

Then perhaps it should be private?

[astafan8]

Agreed - made it private (_connect_to_sqlite_file) in 198f919. It's only used internally by connect() and connect_to_raw_data_db().

[jenshnielsen]

replace this type with a named tuple or dataclass?

[astafan8]

Done in 198f919. Now returns a RawDataDatasetRecord dataclass instead of the opaque tuple.

[jenshnielsen]

@astafan8 That change never seems to have made it to this branch

[astafan8]

and idea: why don't we leverage the export-info infra, add another export type qcodes-raw-sqlite-db with .db extension isntead of adding a new raw_data_db_path column. what would be the pros/cons of that?

[astafan8]

an idea: should this implementation be part of DataSet class or should we add DataSetInSeparateSqliteDbFile class, or a DataSetInSeparateFile class with sqlite-db implementation and a place for further storage implementations? what would be pros-cons of those?

[astafan8]

Re: reusing the export-info infra (a qcodes-raw-sqlite-db export type) instead of a raw_data_db_path column

I researched this and I'd recommend against it. Findings:

1. Semantic mismatch. export_info/export_paths is a registry of post-acquisition exported snapshots, produced by the one-shot DataSet.export() flow (to_xarray/to_pandas → write a file → record the path). Split raw storage instead writes results live during acquisition via add_results/the background writer. It is a storage backend, not an export. Overloading export_info would conflate two genuinely different concepts (and e.g. load_from_exported_file / _get_datasetprotocol_from_guid routing would get tangled between "load an exported netcdf snapshot" and "route live reads to the raw DB").

2. DataExportType is a hard-coded, non-extensible enum.Enum (export_config.py, values nc/csv). Adding qcodes-raw-sqlite-db would require touching the enum and the hard-coded dispatch in BaseDataSet._export_data (if NETCDF / elif CSV / else), and load_from_file (only recognises .nc), and ExportInfo.__post_init__ key validation, and it interacts with the export_type config + set/get_data_export_type. That's more invasive than one dynamic column, with a larger blast radius on the export/config surface.

3. export_info is user-facing metadata. It is stored under the export_info tag in the runs table and appears in DataSet.metadata. In the latest review round we deliberately moved raw_data_db_path out of metadata (into a dedicated column read via get/set_raw_data_db_path_for_run) precisely to stop it leaking into the_same_dataset_as comparisons and into extracted/exported databases. Storing the path inside export_info would put it right back into user-facing metadata and reintroduce those exact bugs.

4. It doesn't even save a schema column. raw_data_db_path is a dynamic column, exactly like export_info — so there's no schema-cleanliness win to be had.

Net: the dedicated, metadata-excluded column is the cleaner model because raw-sqlite storage is a live backend, not an export. If we ever want a unified "where does this dataset's data live" registry, I'd argue for a small purpose-built abstraction rather than overloading export_info.

[astafan8]

Re: should this live in DataSet, or a DataSetInSeparateSqliteDbFile / DataSetInSeparateFile class?

I agree the config-flag-in-DataSet approach is the weakest part architecturally — split behaviour is currently ~15 is_raw_data_storage_enabled() touch-points sprinkled through DataSet. I'm going to implement the subclass option in a separate stacked PR so you can review it as an isolated diff.

Design:

• Introduce a small set of overridable hooks on DataSet, whose base implementations reproduce today's main-DB behaviour:

  • _creates_main_results_table (bool) — base True.
  • _data_conn — base returns self.conn.
  • _setup_results_backend_on_load() / _setup_results_backend_on_new_run() / _setup_results_backend_on_start(paramspecs) — base no-ops.
  • _augment_background_write_item(item) — base no-op.
    The generic parts already added for the no-empty-table work (_results_table_exists, tolerant number_of_results/__len__, deferred subscribers) stay in the base — they are backend-agnostic.

• Add DataSetInSeparateSqliteDbFile(DataSet) that overrides only those hooks (create/connect the per-dataset SQLite file, route _data_conn, record raw_data_db_path). All the is_raw_data_storage_enabled() conditionals then leave DataSet.

• Move the class-selection to the factory boundary:

  • new_data_set() and Measurement instantiate the subclass when the feature is enabled.
  • _get_datasetprotocol_from_guid instantiates the subclass when the run has a raw_data_db_path (this is the disambiguation we already added).

This keeps DataSet backend-agnostic and gives us the natural extension point for the future non-SQLite backends you mentioned: a later DataSetInSeparateFile base could sit between DataSet and DataSetInSeparateSqliteDbFile, with sibling implementations (parquet/zarr/…). I'll start with the concrete SQLite subclass and leave that generalisation as a clean follow-up rather than speculatively abstracting now.

I'll open the stacked PR against this branch shortly.