[multiple-commpaths] Allows colon-separated commpath list by rountree · Pull Request #196 · llnl/Spindle

rountree · 2026-07-06T02:45:01Z

Per-server selection, first path in --commpaths that works for that server gets picked. No consensus process, as the commpath is not visible outside of the server and its particular clients.

…a FE message

Replaces --commpath with --commpaths. Takes a colon-separated list of commpaths. The choice of which of those to use is made per-server. The first workable path is selected. The paths may differ from server to server: unlike --cachepaths, there is no consensus process.

mplegendre · 2026-07-28T14:22:58Z

@rountree - Are you working on fixing the CI failures?

rountree · 2026-08-03T20:47:38Z

@mplegendre Yes. What's queued up for this coming Friday is:

A PR adding workflows for granular resource manager testing and log recovery via artifacts.
A FAQ on how to launch GitHub workflows, monitor progress, and download artifacts all via the command line.
A PR with most of the low-hanging-fruit configure file changes.
A patched multiple-commpaths PR that passes repeated invocations of the existing ci.yml workflow.

See ci-artifacts branch/PR for description. The workflows need be added here as the files are required to be both in the default branch as well as the branch using the workflows.

rountree · 2026-08-07T13:58:47Z

Debugging notes:

Running bd856fa (tip of multiple-commpaths at the moment)
3/13 ci.yml failures
a) Flux, Running: ./run_driver --dependency --push, Error: The operation was canceled. 17m 27s, 31038941417
b) slurm-rshlaunch, SOME TESTS FAILED (no failures logged), 2m 9s, 31038944077
c) slurm-srun, SOME TESTS FAILED (no failures logged), 1m 18s, 31038929449

Running c0170276 (tip of devel)
3/31 ci.yml
a) Flux, Running: ./run_driver --dependency --push, Error: The operation was canceled., 17m 7s, 31043385441
b) ditto
c) ditto

Running bd856fa (multiple-commpaths)
0/31 ci-slum-srun-artifacts failures
0/31 ci-slurm-artifacts failures

Running bd856fa (multiple-commpaths)
4/13 ci.yml failures, 3 Flux, 1 Run spindle-slurm-ubuntu testsuite

Running c0445ae (testing-multiple-commpaths)
Run 31106514887 job 92632975191 "Slurm srun, Ubuntu"

At this point GitHub actions became unusably slow.

The good news is that the podman port can replicate the problem on rzadams with a 4% failure rate.

nchaimov · 2026-08-10T18:21:42Z

+        continue-on-error: true
+        run: |
+          failed=0
+          for node in slurm-head slurm-node-1 slurm-node-2 slurm-node-3 slurm-node-4; do


Suggested change

for node in slurm-head slurm-node-1 slurm-node-2 slurm-node-3 slurm-node-4; do

for node in slurm-plugin-head slurm-plugin-node-1 slurm-plugin-node-2 slurm-plugin-node-3 slurm-plugin-node-4; do

I think this will need to use the container name as defined in the docker-compose.yml, which for the plugin variant includes -plugin- in the name (see for example docker exec slurm-plugin-head above)

nchaimov · 2026-08-10T18:24:52Z


 static char* exitSocketPath(spindle_args_t *params)
 {
   char *realized_dir, *socket_path = NULL;


Suggested change

char *realized_dir = NULL, *socket_path = NULL;

With the change below, if getFirstValidPath returns -1, we goto done without ever assigning to realized_dir; before these changes, we would always assign to realized_dir. Since the done block checks the value of realized_dir, we need to initialize it to NULL.

nchaimov · 2026-08-10T18:27:47Z

   snprintf(options_str, 32, "%lu", (unsigned long) params->opts);
   string options(options_str);

+   getFirstValidPath( params->commpaths, &( params->commpath ), params->number );


Need to handle the case where getFirstValidPath returns -1

nchaimov · 2026-08-10T18:29:00Z

-            [COMMPATH=${withval}],
-            [COMMPATH=$DEFAULT_LOC])
+AC_ARG_WITH(commpaths,
+            [AS_HELP_STRING([--with-compaths=DIR],[Colon-separated list of candidate paths for back-end communication and housekeeping])],


Suggested change

[AS_HELP_STRING([--with-compaths=DIR],[Colon-separated list of candidate paths for back-end communication and housekeeping])],

[AS_HELP_STRING([--with-commpaths=DIR],[Colon-separated list of candidate paths for back-end communication and housekeeping])],

The name of the argument is commpaths but the help string is --with-compaths, with one m instead of two.

nchaimov · 2026-08-10T18:33:59Z

-#if !defined(COMMPATH)
-#error COMMPATH must be defined in config.h
-#endif
+#define SESSIONPATH "/tmp/spindle/session"


I'm not sure we should hard-code a path for the session directory

nchaimov · 2026-08-10T18:34:41Z

-      return NULL;
-   }
-
-   result = spindle_mkdir(dir);


The old code called spindle_mkdir to ensure that the session directory exists, but the new code doesn't. Do we still need the spindle_mkdir?

nchaimov · 2026-08-10T18:36:02Z

-            [AS_HELP_STRING([--with-compath=DIR],[Back-end directory for communication and housekeeping])],
-            [COMMPATH=${withval}],
-            [COMMPATH=$DEFAULT_LOC])
+AC_ARG_WITH(commpaths,


We might want to still accept the old commpath (singular) form so that existing configure arguments continue to work

nchaimov · 2026-08-10T18:52:34Z

The above line increased the size of the buffer, but this line still null-terminates at the original size; is that correct?

rountree temporarily deployed to Spindle CI July 6, 2026 02:45 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 6, 2026 02:45 — with GitHub Actions Failure

rountree had a problem deploying to Spindle CI July 6, 2026 02:45 — with GitHub Actions Error

rountree temporarily deployed to Spindle CI July 6, 2026 02:45 — with GitHub Actions Inactive

rountree requested a review from mplegendre July 6, 2026 02:45

rountree had a problem deploying to Spindle CI July 6, 2026 04:02 — with GitHub Actions Failure

rountree temporarily deployed to Spindle CI July 6, 2026 04:02 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 6, 2026 04:02 — with GitHub Actions Error

rountree temporarily deployed to Spindle CI July 6, 2026 04:02 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 6, 2026 04:02 — with GitHub Actions Error

rountree temporarily deployed to Spindle CI July 6, 2026 04:16 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 6, 2026 04:16 — with GitHub Actions Error

rountree had a problem deploying to Spindle CI July 6, 2026 04:16 — with GitHub Actions Failure

rountree temporarily deployed to Spindle CI July 6, 2026 04:16 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 6, 2026 04:44 — with GitHub Actions Failure

rountree had a problem deploying to Spindle CI July 6, 2026 04:44 — with GitHub Actions Error

rountree force-pushed the multiple-commpaths branch 2 times, most recently from 2d223fe to 8c7c63f Compare July 10, 2026 06:37

rountree temporarily deployed to Spindle CI July 10, 2026 06:38 — with GitHub Actions Inactive

rountree temporarily deployed to Spindle CI July 10, 2026 16:08 — with GitHub Actions Inactive

rountree temporarily deployed to Spindle CI July 10, 2026 16:19 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 10, 2026 16:19 — with GitHub Actions Failure

rountree had a problem deploying to Spindle CI July 10, 2026 16:19 — with GitHub Actions Error

rountree temporarily deployed to Spindle CI July 10, 2026 16:25 — with GitHub Actions Inactive

mplegendre and others added 2 commits July 10, 2026 09:53

Move cachpath consensus to be calculated at init time rather than on …

68bacdf

…a FE message

rountree force-pushed the multiple-commpaths branch from b184a51 to 1072473 Compare July 10, 2026 16:56

rountree temporarily deployed to Spindle CI July 10, 2026 16:56 — with GitHub Actions Inactive

rountree had a problem deploying to Spindle CI July 10, 2026 16:56 — with GitHub Actions Failure

rountree force-pushed the multiple-commpaths branch from 07e1a40 to 1072473 Compare August 1, 2026 02:41

[multiple-commpaths] Adds debugging workflows.

bd856fa

See ci-artifacts branch/PR for description. The workflows need be added here as the files are required to be both in the default branch as well as the branch using the workflows.

rountree force-pushed the multiple-commpaths branch from 44a204f to bd856fa Compare August 5, 2026 17:49

nchaimov self-requested a review August 7, 2026 17:32

nchaimov reviewed Aug 10, 2026

View reviewed changes

Navigation Menu

Uh oh!

[multiple-commpaths] Allows colon-separated commpath list - #196

[multiple-commpaths] Allows colon-separated commpath list#196
rountree wants to merge 3 commits into
llnl:develfrom
rountree:multiple-commpaths

rountree commented Jul 6, 2026

Uh oh!

mplegendre commented Jul 28, 2026

Uh oh!

rountree commented Aug 3, 2026

Uh oh!

rountree commented Aug 7, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

nchaimov Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	for node in slurm-head slurm-node-1 slurm-node-2 slurm-node-3 slurm-node-4; do
	for node in slurm-plugin-head slurm-plugin-node-1 slurm-plugin-node-2 slurm-plugin-node-3 slurm-plugin-node-4; do

	[AS_HELP_STRING([--with-compaths=DIR],[Colon-separated list of candidate paths for back-end communication and housekeeping])],
	[AS_HELP_STRING([--with-commpaths=DIR],[Colon-separated list of candidate paths for back-end communication and housekeeping])],

Uh oh!

Conversation

rountree commented Jul 6, 2026

Uh oh!

mplegendre commented Jul 28, 2026

Uh oh!

rountree commented Aug 3, 2026

Uh oh!

rountree commented Aug 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants