Usage errors for unsupported runs on several MPI ranks

CHANGELOG.md

@@ -14,6 +14,11 @@ SPDX-License-Identifier: CC-BY-4.0
 * `--numba-threads` is now checked against the number of threads Numba accepts, i.e. the CPUs the rank may run on (CPU affinity from MPI binding or the Slurm allocation, e.g. `--cpus-per-task`) or the `NUMBA_NUM_THREADS` environment variable. Exceeding it stops EBFM with an error explaining the cause, instead of a Numba `ValueError` traceback. Exceeding the suggested thread count for the ranks sharing a node is now only a warning; the suggestion no longer divides the CPUs of the whole node by the number of EBFM ranks. https://github.com/EBFMorg/EBFM/pull/190
 * A fatal error now ends EBFM with exit status 1 whenever mpi4py is installed, both standalone and under `mpirun`. EBFM previously aborted MPI with error code 0, so the run, and `mpirun` with it, reported success after logging "Fatal Error in EBFM". The same applies to an MPI handshake version mismatch. A fatal error before the MPI communicator splitting no longer leaves the other components of an MPMD run waiting forever. https://github.com/EBFMorg/EBFM/pull/191
 * Add an example for `--netcdf-dem-mesh-unstructured`: surface elevation and ice thickness from Elmer/Ice XIOS output for the Greenland mesh (`examples/greenland_mesh_v0/xios_dem.nc`). `examples/README.md` describes the contents EBFM expects in such a file. https://github.com/EBFMorg/EBFM/pull/198
+* Add safeguards for illegal MPI runs. https://github.com/EBFMorg/EBFM/pull/201
+    * EBFM on several MPI ranks now stops at startup with an error unless it runs on a partitioned Elmer mesh (`--is-partitioned-elmer-mesh`), one partition per rank. With `--matlab-mesh`, `--netcdf-mesh`, or `--elmer-mesh` without `--is-partitioned-elmer-mesh`, every rank previously computed the whole grid, and the run failed later, e.g. while the ranks wrote the same output file.
+    * EBFM on several MPI ranks now stops at startup with an error if `--use-part` is given. Every rank previously computed the same partition.
+    * EBFM on several MPI ranks now stops at startup with an error if forcing files (`--forcing-dir`) or restart files (`--restart-dir`, `--restart-init`) are used, also on a partitioned Elmer mesh. On a partitioned mesh, `--forcing-dir` previously failed when reading the first forcing file, `--restart-init` failed at the shape check of the restart file, and `--restart-dir` finished without writing a restart file, with a warning at the end. EBFM on a single rank, e.g. as one component of a coupled MPMD run, is unaffected.
+    * EBFM started by an MPI launcher on several processes (e.g. `mpirun -np 4 ebfm ...`) without mpi4py now stops with an error asking to install `ebfm[mpi]`. Every process previously ran the whole model on its own, and the processes then failed while writing the same output file. Detected via the launcher's `OMPI_COMM_WORLD_SIZE` (Open MPI) or `PMI_SIZE` (MPICH, Intel MPI, `srun --mpi=pmi2`).
 
 # v0.8.0
 

README.md

@@ -216,9 +216,17 @@ path/to/your/elmer/mesh
 Usage example for partitioned mesh:
 
 ```sh
-ebfm --elmer-mesh examples/greenland_mesh_v0/MESH/partitioning.128/ --netcdf-dem-mesh examples/BedMachineGreenland-v5_lo.nc --is-partitioned-elmer-mesh --use-part 42
+ebfm --elmer-mesh examples/greenland_mesh_v0/MESH/partitioning.128/ --netcdf-dem-mesh examples/BedMachineGreenland-v5_lo.nc --is-partitioned-elmer-mesh --use-part 42 --elmer-mesh-crs-epsg 3413
 ```
 
+On several MPI ranks, `--use-part` is not allowed: rank i computes the partition i + 1, e.g. the partitions 1 to 4 with:
+
+```sh
+mpirun -n 4 ebfm --elmer-mesh examples/greenland_mesh_v0/MESH/partitioning.128/ --netcdf-dem-mesh examples/BedMachineGreenland-v5_lo.nc --is-partitioned-elmer-mesh --elmer-mesh-crs-epsg 3413
+```
+
+A partitioned mesh is the only grid EBFM runs on several MPI ranks.
+
 Note: The partitioned mesh is not included in this repository.
 
 ### Getting the example data

src/ebfm/core/cli.py

@@ -12,6 +12,7 @@
 """
 
 import argparse
+import os
 from argparse import ArgumentParser, Namespace
 from datetime import datetime
 from enum import Enum
@@ -20,6 +21,7 @@
 import isodate
 
 import ebfm.core
+import ebfm.core.comm
 from ebfm.core import INIT
 from ebfm.core.comm import mpi_available
 from ebfm.core.config import Calendar, FieldValidationLevel, GridConfig, DEFAULT_TZ, iso8601
@@ -38,6 +40,19 @@ def _cli_error(msg: str) -> None:
     _parser.error(msg)
 
 
+def mpi_launcher_size() -> int:
+    """Number of processes an MPI launcher started, read from the environment it sets; 1 if none is found.
+
+    Covers Open MPI's mpirun (OMPI_COMM_WORLD_SIZE) and PMI-based launchers such as MPICH's mpiexec, Intel MPI and
+    srun --mpi=pmi2 (PMI_SIZE). Works without mpi4py.
+    """
+    for variable in ("OMPI_COMM_WORLD_SIZE", "PMI_SIZE"):
+        value = os.environ.get(variable, "")
+        if value.isdigit():
+            return int(value)
+    return 1
+
+
 class CliDefaults(Enum):
     START_TIME = datetime(1979, 1, 1, 0, 0, tzinfo=DEFAULT_TZ)
     END_TIME = datetime(1979, 1, 2, 0, 0, tzinfo=DEFAULT_TZ)
@@ -331,7 +346,7 @@ def parse_time_step(value: str) -> float | str:
         type=int,
         default=argparse.SUPPRESS,  # To not print (default: ...) in --help
         help="If using a partitioned Elmer mesh, allows to specify which partition ID to use for this run. "
-        "If not provided, the MPI rank + 1 will be used as partition ID.",
+        "If not provided, the MPI rank + 1 will be used as partition ID. Only allowed in runs on a single MPI rank.",
     )
 
     logger_group = parser.add_argument_group("logging configuration")
@@ -471,6 +486,14 @@ def parse_time_step(value: str) -> float | str:
     if active_mpi_opts and not mpi_available:
         parser.error(f"{', '.join(active_mpi_opts)} require MPI. Install via: pip install 'ebfm[mpi]'")
 
+    n_launched = mpi_launcher_size()
+    if n_launched > 1 and not mpi_available:
+        parser.error(
+            f"EBFM was started on {n_launched} MPI processes, but mpi4py cannot be imported "
+            f"({getattr(ebfm.core.comm, 'mpi_import_error', 'not installed')}), so every process would run the whole "
+            "model on its own. Install via: pip install 'ebfm[mpi]'"
+        )
+
     return args
 
 
@@ -504,3 +527,75 @@ def validate_shading_coupling_compat(grid_config, coupling_config) -> None:
             "Shading routine not implemented for coupled runs. "
             "Please deactivate shading via --no-shading or deactivate coupling."
         )
+
+
+def validate_grid_parallel_compat(grid_config, comm) -> None:
+    """Check that a run on several MPI ranks uses a partitioned Elmer mesh, of which each rank computes one partition.
+
+    Must be called on every rank of comm.
+
+    @param[in] grid_config resolved GridConfig
+    @param[in] comm        EBFM's MPI communicator
+    """
+    if comm.size == 1:
+        return
+
+    if grid_config.grid_type is GridInputType.MATLAB:
+        rejected_grid = "--matlab-mesh"
+    elif grid_config.grid_type is GridInputType.NETCDF:
+        rejected_grid = "--netcdf-mesh"
+    elif not grid_config.is_partitioned:
+        rejected_grid = "--elmer-mesh without --is-partitioned-elmer-mesh"
+    else:
+        return
+
+    _cli_error(
+        f"{rejected_grid} is not supported for runs on several MPI ranks ({comm.size} ranks). "
+        "Only a partitioned Elmer mesh (--is-partitioned-elmer-mesh) runs on several ranks, one partition per rank. "
+        "Please run on a single rank."
+    )
+
+
+def validate_use_part_parallel_compat(args: Namespace, comm) -> None:
+    """Check that --use-part is not given in a run on several MPI ranks, where every rank would compute it.
+
+    Must be called on every rank of comm, before args.use_part defaults to the rank's partition.
+
+    @param[in] args parsed CLI arguments
+    @param[in] comm EBFM's MPI communicator
+    """
+    if hasattr(args, "use_part") and comm.size > 1:
+        _cli_error(
+            f"--use-part is not supported for runs on several MPI ranks ({comm.size} ranks), since every rank would "
+            f"compute partition {args.use_part}. Please leave out --use-part: rank i then computes partition i + 1."
+        )
+
+
+def validate_forcing_parallel_compat(args: Namespace, comm) -> None:
+    """Check that no forcing files are read in a run on several MPI ranks.
+
+    Must be called on every rank of comm.
+
+    @param[in] args parsed CLI arguments
+    @param[in] comm EBFM's MPI communicator
+    """
+    if args.forcing_dir is not None and comm.size > 1:
+        _cli_error(
+            f"Forcing files (--forcing-dir) are not supported for runs on several MPI ranks ({comm.size} ranks). "
+            "Please run on a single rank."
+        )
+
+
+def validate_restart_parallel_compat(args: Namespace, comm) -> None:
+    """Check that no restart files are read or written in a run on several MPI ranks.
+
+    Must be called on every rank of comm.
+
+    @param[in] args parsed CLI arguments
+    @param[in] comm EBFM's MPI communicator
+    """
+    if (args.restart_dir is not None or args.restart_init) and comm.size > 1:
+        _cli_error(
+            f"Restart files (--restart-dir, --restart-init) are not supported for runs on several MPI ranks "
+            f"({comm.size} ranks). Please run on a single rank."
+        )

src/ebfm/main.py

@@ -25,6 +25,10 @@
     parse_cli_args,
     resolve_numba_threads,
     validate_shading_coupling_compat,
+    validate_grid_parallel_compat,
+    validate_use_part_parallel_compat,
+    validate_forcing_parallel_compat,
+    validate_restart_parallel_compat,
 )
 
 import ebfm.coupling
@@ -138,6 +142,9 @@ def _main_impl():
 
     forcing_config = ForcingConfig(coupling_config, args)
 
+    # Ensure --use-part is only used in runs on a single rank.
+    validate_use_part_parallel_compat(args, ebfm_comm)
+
     if not hasattr(args, "use_part"):
         # If not provided via command line option --use-part, set to rank + 1 (assuming partition IDs start at 1).
         args.use_part = ebfm_comm.rank + 1
@@ -172,6 +179,15 @@ def _main_impl():
     # See https://github.com/EBFMorg/EBFM/issues/11 for details.
     validate_shading_coupling_compat(grid_config, coupling_config)
 
+    # Ensure a run on several ranks uses a partitioned mesh.
+    validate_grid_parallel_compat(grid_config, ebfm_comm)
+
+    # Ensure forcing files are only read in runs on a single rank.
+    validate_forcing_parallel_compat(args, ebfm_comm)
+
+    # Ensure restart files are only read and written in runs on a single rank.
+    validate_restart_parallel_compat(args, ebfm_comm)
+
     logger.debug("Successfully completed consistency checks.")
 
     # Model setup & initialization