Repository navigation
Usage errors for unsupported runs on several MPI ranks - #201
Merged
Merged
Conversation
On several MPI ranks, EBFM only runs on a pre-partitioned Elmer mesh, one partition per rank. With any other grid, every rank computed the whole grid and the run failed late, e.g. with the ranks writing the same output file. On a partitioned mesh, --forcing-dir failed at the first forcing read, --restart-init at the restart shape check, and --restart-dir finished without a restart file. These are now usage errors, raised before INIT reads the grid. The checks use EBFM's own communicator, so EBFM on a single rank of a coupled MPMD run is unaffected. Assisted-by: Claude Code:claude-opus-5-5
Without mpi4py, every process started by mpirun ran as a serial run over the whole grid, and the processes then failed while writing the same output file with a misleading PermissionError. EBFM now reads the process count the launcher puts into the environment, OMPI_COMM_WORLD_SIZE (Open MPI) or PMI_SIZE (MPICH, Intel MPI, srun --mpi=pmi2), and stops with a usage error if it is above 1 and mpi4py cannot be imported. Serial runs without mpi4py are unaffected. Assisted-by: Claude Code:claude-opus-5-5
BenjaminRodenberg
force-pushed
the
mpi-usage-errors
branch
from
October 21, 2026 10:31
1d3d168 to
c02453e
Compare
--use-part sets the partition of every rank, so on several ranks all of them computed the same partition. It is now a usage error there; without it, rank i computes partition i + 1 as before. A single rank, e.g. EBFM in a coupled MPMD run, may still pick its partition. The README example for a partitioned mesh now passes the required --elmer-mesh-crs-epsg and shows a run on several ranks. Assisted-by: Claude Code:claude-opus-5-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extracted from #197 as a small, independent PR.
On main, EBFM on several MPI ranks only works with a pre-partitioned Elmer mesh (
--elmer-mesh .../partitioning.N --netcdf-dem-mesh ... --is-partitioned-elmer-mesh, one partition per rank). Every other setup runs the whole grid on every rank and fails late with errors that don't point at the cause; e.g.mpirun -np 2 ebfm --matlab-mesh examples/dem_and_mask.matfails in the first time step with aPermissionErrorwhile both ranks write the same output file. This PR turns these setups into usage errors at startup (exit status 2). It is purely a usability change: no model change and no new parallel capability.Rejected when EBFM's own communicator has more than one rank, before INIT reads the grid:
--matlab-mesh,--netcdf-mesh, and--elmer-meshwithout--is-partitioned-elmer-mesh(validate_grid_parallel_compat)--forcing-dir(validate_forcing_parallel_compat, from 7a1a987)--restart-dir/--restart-init, also on a partitioned mesh (validate_restart_parallel_compat, from 07b14b0)Rejected at parse time, independently of the communicator: an MPI launcher started EBFM on several processes (
OMPI_COMM_WORLD_SIZEorPMI_SIZEabove 1), but mpi4py cannot be imported (from 912b160).The guards check
ebfm_comm, notCOMM_WORLD, so a coupled MPMD run with EBFM on one rank (README:mpirun -np 1 ebfm ... --is-partitioned-elmer-mesh --use-part 1 : -np 1 elmer_dummy_f.x ...) is unaffected, restart files included. Pre-partitioned runs on several ranks work as on main, except:--forcing-diralready failed when reading the first forcing file; it now fails at startup.--restart-dirused to finish without a restart file, with a warning at the end; it now fails at startup.--restart-initalready failed at the shape check of the restart file; it now fails at startup.Nothing else from #197 is included. The MATLAB part of #197 (branch
mpi-matlab-mesh) builds on this PR: it lifts the--matlab-meshrestriction and adds the shading guard, which cannot be reached until then.Known issue, not changed here: a usage error raised after the communicator splitting in an MPMD job leaves the other components waiting. With a partner that joins the MPI handshake and then waits in a barrier on the coupler group,
mpirun -np 2 ebfm --matlab-mesh ... : -np 1 partnerhangs after both EBFM ranks print the usage error (Open MPI 4.1.6). The existingvalidate_shading_coupling_compatraises at the same point and hangs the same way on main.Tested:
pytest tests: 197 passed (mpi4py 4.1.2, Open MPI 4.1.6)mpirun -np 2 python -m ebfm.main --matlab-mesh examples/dem_and_mask.mat: usage error on both ranks, exit status 2, no hangpartitioning.128) on 2 ranks: runs as on main; with--forcing-dir,--restart-dirand--restart-init, main behaves as listed above and this branch stops at startupmpirun -np 2with mpi4py not importable: usage error on both processes, exit status 2; a serial run without mpi4py is unaffectedGenerated with Claude Code