Repository navigation
Read a NetCDF DEM in bands instead of as a whole - #200
Open
BenjaminRodenberg wants to merge 7 commits into
Open
BenjaminRodenberg wants to merge 7 commits into
BenjaminRodenberg wants to merge 7 commits into
Conversation
BenjaminRodenberg
force-pushed
the
dem-banded-read
branch
from
October 20, 2026 18:54
71ed369 to
f5a384a
Compare
Collaborator
Author
setup-mpi makes Open MPI use TCP between ranks, and on GitHub's runners Open MPI's TCP transport also picks the docker0 interface, over which messages never arrive. Small messages still get through, but large ones hang: in #197 a gather on 4 ranks hung until GitHub's limit of 6 hours, while the same run passed locally, where the ranks use shared memory. The next commits run the MATLAB example on up to 4 ranks, with gathers large enough to hang. Open MPI now only uses the loopback interface in both workflows. It is set in mca-params.conf, since tox does not pass OMPI_MCA_* variables to the tests. In addition, the build-and-test job now stops after 20 minutes and its MPI steps after 5, so a hang fails quickly instead of blocking a runner for 6 hours. Taken from 5e28f4d and 47b4c8f of #197. Assisted-by: Claude Code:claude-opus-5-5
MATLAB and NetCDF grids now keep the flat indices of their columns in the 2-D fields as `ind` and take every per-column field from the 2-D fields with them, instead of applying the mask to each field again. A subset of the columns then only needs a subset of `ind`, which the next commits use to split a MATLAB grid over MPI ranks. The columns, their order and the model output are unchanged. Assisted-by: Claude Code:claude-opus-5-5
The output writer now gathers the columns of all ranks with ColumnGather and rank 0 writes them to the file a single rank would write, one variable at a time to limit its memory. Only structured grids are gathered; a partitioned mesh still skips the output file. Taken from 8b5a9eb of #197. Assisted-by: Claude Code:claude-opus-5-5
The shading look-up table is computed by walking from every glacier cell of the 2-D grid, not only from the columns of one rank, so a rank holding part of the columns cannot build its rows of the table. Shading on several ranks is now a usage error asking for --no-shading, and _precompute_shading_matlab asserts that it is not called in such a run. No grid that supports shading runs on several ranks yet; the next commits split MATLAB grids over the ranks. Taken from 43a676e of #197. Assisted-by: Claude Code:claude-opus-5-5
On several ranks, each rank now computes a contiguous block of the columns of a MATLAB grid, the first block on rank 0. The 2-D fields stay whole on every rank: the slope needs the neighbours of each cell, and the output writer places the columns of all ranks into them. The domain-wide latitude mean, lapse rate and random seed of #203 make the results equal those of a single rank. Taken from 46f47a2 of #197. Assisted-by: Claude Code:claude-opus-5-5
The MATLAB example runs for one day on a single rank and on 2, 3 and 4 ranks under mpirun; each parallel run must give a byte-identical output file and an equal --dump-reference snapshot. CI also runs the MATLAB example on 4 ranks. Taken from db2b582 and e26d3e8 of #197. Assisted-by: Claude Code:claude-opus-5-5
read_dem loaded the whole surface variable, about 750 MB for BedMachineGreenland-v5.nc, before sampling it at the cell centres. It now reads bands of at most 4M values, and of each band only the window around the points it samples. A single bounding box does not help: the cells of the Greenland mesh span nearly the whole DEM. Peak RSS after grid initialization on the Greenland mesh with the full-resolution DEM drops from about 1.1 GB to about 0.3 GB. Results are unchanged. Assisted-by: Claude Code:claude-opus-5-5
BenjaminRodenberg
force-pushed
the
dem-banded-read
branch
from
October 21, 2026 16:08
f5a384a to
87d998a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
read_dem loaded the whole surface variable, about 750 MB for BedMachineGreenland-v5.nc, before sampling it at the cell centres. It now reads bands of at most 4M values, and of each band only the window around the points it samples. A single bounding box does not help: the cells of the Greenland mesh span nearly the whole DEM.
Peak RSS after grid initialization on the Greenland mesh with the full-resolution DEM drops from about 1.1 GB to about 0.3 GB. Results are unchanged.
Assisted-by: Claude Code:claude-opus-5-5
Author's Todos