Widen some dtypes to avoid overflows in SSC logic trees with many src_groups - #11709
Open
CB-quakemodel wants to merge 8 commits into
Open
Widen some dtypes to avoid overflows in SSC logic trees with many src_groups#11709CB-quakemodel wants to merge 8 commits into
CB-quakemodel wants to merge 8 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change 1
Widen
perf_dt.task_nofromint16toint32so calcs submitting more than 32,767 tasks no longer overflow to provide an error ofOverflowError: Python integer 32768 out of bounds for int16This was observed when running the complex BC Hydro NVA model (many source groups) - it occurs if using seemingly about 20,000 samples or more, with either the regular or proposed sequential approach.
Change 2
Widen
gid/grp_id, which is the (source_group, gsim) label in the_ratestable fromU16toU32. This change is made forrates_dt,source_info_dt,rup/grp_id, and thestore_csmdtype.s The "old" datastores (i.e., still useU16for these dtypes) remain readable.This was required because
gidcolumn dtype is currentlyU16, capping gids at 65,536. For the very large logic tree of BC Hydro NVA (again many source groups at N ≥ 16,000 samples), rows withgid ≥ 65,536are being silently dropped, so realisations mapping to those high gids get zero rate. This manifests in the 5th/16th percentile curves at each site collapsing to zero in the BC Hydro NVA model when using 16,000 samples (but not when using say only 1000).