Skip to content

Widen some dtypes to avoid overflows in SSC logic trees with many src_groups - #11709

Open
CB-quakemodel wants to merge 8 commits into
masterfrom
widen_task_no_dtype
Open

Widen some dtypes to avoid overflows in SSC logic trees with many src_groups#11709
CB-quakemodel wants to merge 8 commits into
masterfrom
widen_task_no_dtype

Conversation

@CB-quakemodel

@CB-quakemodel CB-quakemodel commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Change 1

Widen perf_dt.task_no from int16 to int32 so calcs submitting more than 32,767 tasks no longer overflow to provide an error of OverflowError: Python integer 32768 out of bounds for int16

This was observed when running the complex BC Hydro NVA model (many source groups) - it occurs if using seemingly about 20,000 samples or more, with either the regular or proposed sequential approach.

Change 2

Widen gid/grp_id, which is the (source_group, gsim) label in the _rates table from U16 to U32. This change is made for rates_dt, source_info_dt, rup/grp_id, and the store_csm dtype.s The "old" datastores (i.e., still use U16 for these dtypes) remain readable.

This was required because gid column dtype is currently U16, capping gids at 65,536. For the very large logic tree of BC Hydro NVA (again many source groups at N ≥ 16,000 samples), rows with gid ≥ 65,536 are being silently dropped, so realisations mapping to those high gids get zero rate. This manifests in the 5th/16th percentile curves at each site collapsing to zero in the BC Hydro NVA model when using 16,000 samples (but not when using say only 1000).

@CB-quakemodel CB-quakemodel changed the title Widen task_no field dtype to 32 bit Widen some dtypes to avoid overflows in SSC logic trees with many src_groups Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant