I found in the course of testing that if I run the kvcache benchmark after installing and running the vectordb tests that a hang will occur at the beginning when the benchmark is setting up the collector. Note that this only occurs on multiple client runs, running with a single client doesn't show this behavior. I have a workaround but wanted to bring this up so it could get addressed in future versions or in the docs.
In this condition, kcking of kvcache with 2 or more hosts will get to the following line and hang, ultimately timing out and failing.
CLI: uv run mlpstorage open kvcache run --hosts wsclient01 wsclient02 --cache-dir /mnt/kv --results-dir /mnt/logs/0811
Collecting via MPI... 0:00:002026-08-11 13:51:58|INFO:cluster_collector:3366: Collector script staged on wsclient02:/mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging
2026-08-11 13:51:58|INFO:cluster_collector:3457: Running MPI collection across 2 host(s)
I ran with --debug and saw that the hang happened during this mpirun command:
mpirun -n 2 -host wsclient01:1,wsclient02:1 --bind-to none --map-by node /home/nutanix/mlperf3/storage/.venv/bin/python3 /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/mlps_collector.py /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/cluster_info.json
which hung and showed this warning message:
WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.
This attempted connection will be ignored; your MPI job may or may not
continue properly.
Local host: wsclient02
PID: 6181
Searching around this is likely caused by a multi-home issue where MPI is trying to use a particular network interface while the partner listening on a different one.
The only thing generally different about these clients is that I used them for running Vectordb testing before running kvcache so there is a docker0 interface and a docker bridge interface on each one.
After deleting the two interfaces on each client (docker0 and br-xxxxxxxx) the benchmark executed fine.
I found in the course of testing that if I run the kvcache benchmark after installing and running the vectordb tests that a hang will occur at the beginning when the benchmark is setting up the collector. Note that this only occurs on multiple client runs, running with a single client doesn't show this behavior. I have a workaround but wanted to bring this up so it could get addressed in future versions or in the docs.
In this condition, kcking of kvcache with 2 or more hosts will get to the following line and hang, ultimately timing out and failing.
CLI: uv run mlpstorage open kvcache run --hosts wsclient01 wsclient02 --cache-dir /mnt/kv --results-dir /mnt/logs/0811
Collecting via MPI... 0:00:002026-08-11 13:51:58|INFO:cluster_collector:3366: Collector script staged on wsclient02:/mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging
2026-08-11 13:51:58|INFO:cluster_collector:3457: Running MPI collection across 2 host(s)
I ran with --debug and saw that the hang happened during this mpirun command:
mpirun -n 2 -host wsclient01:1,wsclient02:1 --bind-to none --map-by node /home/nutanix/mlperf3/storage/.venv/bin/python3 /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/mlps_collector.py /mnt/logs/0811/open/Nutanix/results/wsclient01/kv_cache/tiny-1b/run/20260811_135054/collector-staging/cluster_info.json
which hung and showed this warning message:
WARNING: Open MPI accepted a TCP connection from what appears to be a
another Open MPI process but cannot find a corresponding process
entry for that peer.
This attempted connection will be ignored; your MPI job may or may not
continue properly.
Local host: wsclient02
PID: 6181
Searching around this is likely caused by a multi-home issue where MPI is trying to use a particular network interface while the partner listening on a different one.
The only thing generally different about these clients is that I used them for running Vectordb testing before running kvcache so there is a docker0 interface and a docker bridge interface on each one.
After deleting the two interfaces on each client (docker0 and br-xxxxxxxx) the benchmark executed fine.