Neighbor Communication with GPI-2

Overview

This small C example exchanges arrays between GASPI processes arranged in a periodic two-dimensional grid. It demonstrates registered memory segments, one-sided notified writes, receiver-side notification handling, queue management, and collective synchronization using GPI-2, an implementation of GASPI.

The communication concepts come from work developed during an internship and are related to the author's university thesis. This repository is an educational extraction of those ideas, not the full thesis implementation or a production framework.

The checked-in program has one application thread per process. Although the original project concerned thread-based message passing, test.c creates no worker threads and contains no pthread or OpenMP calls. Linking pthread does not create application threads.

Build status: the supplied Makefile and launcher contain inconsistent source, target, and executable names. The repository cannot currently be built and launched through the supplied workflow without corrections. See Building and Limitations before trying to run it.

Key Concepts

• High-performance computing (HPC) distributes work across computing resources. Here the focus is communication and data checking, with no numerical solver or application workload.
• Distributed memory: each process has its own address space and rank. A local C pointer cannot address another process's memory. GASPI identifies remotely accessible memory through segment IDs and byte offsets.
• Message passing: conceptually, each rank sends a directional array to a neighbor. Technically, this example uses one-sided writes into the neighbor's registered segment rather than matched send/receive operations.
• MPI and GASPI: both provide distributed parallel programming interfaces. GPI-2 implements GASPI; this application does not call or require MPI. See the comparison below.
• Processes and threads: processes exchange data through GASPI; each process's main thread initializes buffers, issues communication, waits, and checks results. Any internal runtime threads belong to the GPI-2 implementation.
• Asynchrony and synchronization: writes are submitted through queue 0. A receiver waits for a notification before reading the corresponding data. The program immediately waits after each directional submission, so it does not demonstrate communication overlapped with useful computation.

Architecture

All ranks execute the same main() in test.c. Rank 0 additionally prints the run header. Rank coordinates are x = rank % NPIX, y = rank / NPIX; neighbors wrap at grid boundaries. peer_top increases the row coordinate with wrapping; peer_bot decreases it.

Each rank owns eight segments: four send arrays and four receive arrays, each containing NELEM doubles. Segment IDs are identical across ranks, but their storage is separate. Even when neighbors occupy the same machine, the application accesses their storage through GASPI rather than shared C pointers. With a grid dimension of one, a neighbor can be the rank itself; with a dimension of two, opposite neighbors coincide.

flowchart LR
    subgraph L[Left neighbor process]
        LS[Right send segment 2]
    end
    subgraph P[Process at rank r]
        T[Main application thread]
        Q[Local GASPI queue 0]
        S[Right send segment 2]
        R[Right receive segment 3 and notification 0]
        O[Left, top, bottom segment pairs]
        T --> Q
        T --> S
        R --> T
        T --- O
    end
    subgraph N[Right neighbor process]
        NR[Right receive segment 3 and notification 0]
    end
    LS -->|Notified write| R
    S -->|Notified write via queue 0| NR

Loading

This diagram shows the rightward phase; neighbor roles can refer to the same process in small grids. The other paths and the exact synchronization sequence are documented in Communication flow.

Requirements and Dependencies

The files target a Unix-like cluster environment: test.c uses gettimeofday() and Unix headers, and the scripts use Bash and Slurm. The repository provides no tested operating-system matrix or evidence of a known working platform/version combination. Windows and macOS support are not established.

Requirement	Evidence and scope
C compiler and Make	The explicit recipe invokes gcc -Wall -O3; the variadic error macro uses GNU syntax.
GPI-2 development installation	GASPI.h, libGPI2, and the gaspi_run launcher are needed. The script names the site module gpi2/1.3.0; this is not evidence of compatibility with every release.
System libraries	The explicit recipe links libibverbs, libm, and pthread, even though the application itself has no thread API calls.
Include/library paths	GPIINC must name the directory containing GASPI.h; GPILIB must name the GPI-2 library directory for the explicit Makefile recipe.
Runtime search paths	gaspi_run must be on PATH; shared libraries must be discoverable by the loader, potentially through LD_LIBRARY_PATH as described by upstream.
Cluster launch environment	mkrun.sh requires environment modules, sbatch, a shortrun partition, and site-specific node names. Nodes must access the program and runtime files.

The transport and launch requirements depend on the installed GPI-2 configuration. Upstream documents InfiniBand and TCP options, plus passwordless SSH when using its SSH launcher. InfiniBand hardware is therefore not an unconditional GASPI requirement, although this Makefile explicitly links the verbs library. Consult the upstream installation instructions for your installation rather than assuming the site modules exist locally.

Building the Project

1. Install/configure GPI-2 and its development dependencies according to upstream documentation.

2. Identify the actual header and library directories and set GPIINC and GPILIB to those directories. These are inputs to this repository's explicit recipe, not paths defined by this repository.

3. Inspect the supplied recipes without compiling:

   make -n
   make -n test2D

The current Makefile has these verified inconsistencies:

Entry	Actual behavior
all: test	There is no explicit test recipe. GNU Make falls back to its built-in C rule, attempting cc test.c -o test, without the GASPI flags or configuration macros.
test2D: test.c	The recipe compiles test2D.c, which does not exist, and writes test2D.exe. It also names a target different from its output file.
clean	Removes only test.exe, not test or test2D.exe.
Launcher	Invokes default make, then tries to run test.exe, which neither recipe produces.

Consequently there is no verified successful build command for the supplied Makefile. Even with GPI-2 installed, the default rule lacks definitions of NPIX and NPIY, which have no source defaults. Setting GPIINC and GPILIB alone does not fix that rule. No replacement build command is presented as a tested workaround.

Recommended follow-up: agree on one target/executable name, compile the existing test.c with the explicit GASPI flags and macros, and align all, clean, and the launcher with that output. Those changes are intentionally outside this documentation update.

Running the Example

There are no command-line application options; grid size, message length, and iteration count are compile-time settings. Runtime rank count comes from the GASPI launch environment and must match the compiled grid.

mkrun.sh expresses the intended Slurm workflow. It is currently blocked by the build problems above. After those are addressed and the site configuration is reviewed, its entry point is bash mkrun.sh from the repository root. It:

1. Resolves shell parameters, creates a timestamped gpi-test-NPIXxNPIY-* directory, and symlinks the C sources, Makefile, and helper.
2. Purges modules and loads devtoolset-7 gpi2/1.3.0.
3. Runs make clean and make with explicit parameter assignments, exiting if compilation fails.
4. Writes run.slurm with resource requests, including 64000 MB memory, exclusive nodes, and partition shortrun.
5. Submits it with sbatch. The batch script sources printmachinefile, writes machinefile, sets OMP_NUM_THREADS=1, and intends to execute gaspi_run -m machinefile ./test.exe.

The batch script only echoes the module-load command; it relies on the submission environment being available in the job. CC, CFLAGS, and DEF assigned in mkrun.sh are not exported or passed as Make arguments and do not configure its explicit hard-coded recipe. The OpenMP environment variable does not control application threads here.

printmachinefile defines a Bash function rather than running it automatically. It expands Slurm node lists into node%02d hostnames, repeated SLURM_NTASKS / SLURM_JOB_NUM_NODES times per node. It assumes uniform placement and node names such as node[01-02]; it is not a general Slurm hostname parser. For example, this helper-only check needs no scheduler:

bash -c 'source ./printmachinefile; SLURM_JOB_NODELIST="node[01-02]" SLURM_NTASKS=4 SLURM_JOB_NUM_NODES=2 printmachinefile'

It prints node01 twice and node02 twice. The helper changes IFS to a comma in the sourcing shell without restoring it.

Understanding the Implementation

All application logic is in main() in test.c:

1. Initialize: gaspi_proc_init(GASPI_BLOCK) starts the runtime; gaspi_proc_rank and gaspi_proc_num identify the rank and process count. Each rank calculates its four periodic peers. The runtime count is not checked against the grid.
2. Allocate: eight collective gaspi_segment_create calls register initialized memory in GASPI_GROUP_ALL. gaspi_segment_ptr obtains local pointers. Send arrays encode the element index and rank using direction-specific decimal factors; receive arrays start at -1.0.
3. Start together: a group barrier follows initialization. gaspi_queue_size_max obtains the queue capacity.
4. Exchange: for each iteration, submit right, left, then (if NPIY > 1) top and bottom buffer writes. Each gaspi_write_notify transfers the entire send array at offset zero to the same named receive segment on a peer, attaching notification value rank + 1.
5. Wait and validate: gaspi_notify_waitsome waits for the local incoming notification; gaspi_notify_reset clears it and returns its value, which the application does not inspect. After all phases, compare every received element with the expected neighbor's data, reset receive arrays to -1.0, count failed iterations per direction, and enter a barrier before reuse.
6. Report and terminate: a rank-ordered reporting loop uses barriers to give each rank a turn. gaspi_proc_term(GASPI_BLOCK) terminates the runtime. There are no explicit segment deletions or unconditional final queue drain; the code leaves resource teardown to runtime termination. GASPI errors cause SUCCESS_OR_DIE to print the failed call and exit immediately without an orderly cleanup path.

gaspi_queue_size checks occupancy before each submission; the code calls gaspi_wait only when the queue equals its maximum. A notification indicates incoming data readiness; gaspi_wait concerns outgoing queue completion. A barrier coordinates ranks and is not a substitute for queue completion. See the detailed flow for the distinction and the queue-capacity caveat.

Configuration

Make assignments affect the explicit test2D recipe only; the default implicit rule does not use them. They are compile-time definitions, not runtime command-line flags.

Parameter	Makefile default	Meaning in the implementation
NPI	2	Intended total process count in the script. Passed as a macro but unused in test.c; actual count is obtained with gaspi_proc_num.
LNPI	2	Intended processes per node, used for Slurm placement. Passed as a macro but unused in test.c. The Makefile's “MPI-task” comments are historical terminology.
NPIX	1	Grid width used in all peer calculations; no source fallback.
NPIY	2	Grid height; enables vertical exchange/checking only when greater than one; no source fallback.
NITER	1000	Iterations of exchange, validation, and barrier. Source fallback: 1000.
NELEM	32	Doubles per directional array; bytes per write are NELEM * sizeof(double). Source fallback: 1024.

The shell defaults in mkrun.sh differ: NPIX=2, NPIY=2, NITER=100, NELEM=32, NNODES=1, and NPI=NPIX*NPIY unless supplied. It always computes LNPI=NPI/NNODES (or 1 for NPI=1), overriding an inherited LNPI. NCORE defaults to integer 16/LNPI and controls Slurm CPUs per task, not application threads. The warning printed for default NPIY mistakenly says NPIX.

Use positive grid dimensions, NITER, and NELEM, with runtime ranks equal to NPIX*NPIY. Keep allocations and writes within the installed runtime's limits. The script assumes positive NNODES, an evenly divisible rank distribution, and a positive NCORE; none of these constraints is validated. Its hard-coded assumption of 16 CPUs is site-specific. Vertical segments are allocated even when vertical exchanges are disabled.

MPI vs. GPI-2

MPI is a communication standard with multiple implementations; GASPI is another specification, implemented here by GPI-2. Prior MPI experience helps with ranks, topology, distributed storage, and collective participation. No MPI library is used by this example.

Concept	MPI perspective	This GASPI example
Transfer	Two-sided send/receive matches messages using ranks, tags, and communicators. MPI also supports one-sided RMA.	Sender specifies remote segment ID and offset in gaspi_write_notify; receiver does not post a matching receive.
Storage	Ordinary buffers for two-sided transfers; windows expose memory for MPI RMA.	Registered segments expose arrays for remote access.
Completion	Nonblocking MPI operations use requests and completion calls; MPI RMA has its own synchronization rules.	Outgoing queue completion uses gaspi_wait; incoming readiness uses notification wait/reset. These are different responsibilities.
Coordination	Collectives such as MPI_Barrier coordinate a communicator.	gaspi_barrier coordinates GASPI_GROUP_ALL; all ranks follow the same collective sequence.

Two-sided MPI can suit explicit matched messages; one-sided APIs can suit direct updates to known remote memory layouts. Both MPI and GASPI offer asynchronous communication mechanisms. The appropriate choice depends on the algorithm, implementation, transport, and synchronization requirements; this repository establishes no performance advantage. Consult the MPI standard and GASPI resources for their full semantics.

Limitations

• Educational exchange/checking example: no worker-thread messaging, computation overlap, solver, fault recovery, or validated performance study.
• Build and executable mismatches block the supplied workflow. Modules, scheduler requests, hostname parsing, and resource assumptions require site review.
• Grid/rank consistency, allocation sizes, notification limits, and scheduler parameter constraints are not checked. Invalid peers may cause errors or indefinite waits; operations use GASPI_BLOCK throughout.
• Queue checks wait only when completely full. They do not reserve capacity for the combined write/notification operation. Review the installed GASPI queue requirements before relying on this pattern.
• Send arrays remain constant across iterations; notification values are ignored. The example does not establish a general message protocol or detect arbitrary stale messages.
• Timings accumulated with gettimeofday() are microseconds, although output labels them s. The left result prints dT[0] (right phase), and the right result prints dT[1] (left phase). Phase timings include notification waiting and differ in whether queue-management time is included. They are not isolated network latency or bandwidth measurements.
• Failure summaries count iterations with errors, but the return value uses only the last iteration's element-error counts. An earlier failed iteration can therefore still yield a zero exit status.
• No explicit segment deletion or final gaspi_wait is present. Error exits do not perform coordinated termination.

These issues are documented, not fixed, to preserve runtime behavior. Recommended separate fixes are to align the build/launch names, validate configuration, review queue capacity and teardown, correct timing reports and cumulative exit status, and replace site-specific launch assumptions where portability is needed.

Verification Status

During this documentation update, GNU Make dry runs confirmed both recipe paths; an actual default make failed because GASPI.h is unavailable in the review environment. GPIINC and GPILIB were unset and gaspi_run was absent. Bash syntax checks and the helper example above passed. Relative Markdown file links were checked; Mermaid syntax and source correspondence were reviewed manually, with structural checks but no installed Mermaid renderer. No GPI-2 compilation, multi-rank execution, or performance measurement was completed. Launch instructions describe the checked-in workflow and its blockers, not a successful cluster run.

References

• GPI-2 source, installation guidance, and documentation build instructions. Upstream documents make docs for generating its API documentation.
• GASPI specification 16.6 (PDF, GASPI Forum): API definitions for process management, segments, communication queues, notifications, and barriers. Also consult the GPI-2 documentation sources and generated API manual matching your installed release.
• GASPI project: specification and documentation. The GASPI site could not be retrieved during this review; consult it directly for specification downloads.
• MPI Forum standards and documentation, including two-sided communication, nonblocking completion, and RMA.