A seeded solve gives the same bits on any number of threads or ranks; seed 0 is a seed; BERTINI_NUM_THREADS

.github/workflows/build_and_test.yml

@@ -232,52 +232,60 @@ jobs:
           CCACHE_DIR: ${{ github.workspace }}/.ccache
         run: ccache -s || true
 
-      # Run the suite under BOTH thread settings.  OMP_NUM_THREADS overrides the requested per-solve
-      # thread count (parallel::EffectiveThreadCount), so the serial and threaded code paths genuinely
-      # differ; running both on every platform catches threading bugs (e.g. clone-per-thread) that a
-      # single setting would hide.  `--parallel` is ctest's *process* parallelism (independent of the
-      # per-solve OMP count), so each leg still runs the test executables concurrently.
-      - name: Test (threaded + OMP_NUM_THREADS=1)
+      # Run the suite under BOTH thread settings.  BERTINI_NUM_THREADS overrides the requested
+      # per-solve thread count (parallel::EffectiveThreadCount), so the serial and threaded code paths
+      # genuinely differ; running both on every platform catches threading bugs (e.g. clone-per-thread)
+      # that a single setting would hide.  `--parallel` is ctest's *process* parallelism (independent
+      # of the per-solve thread count), so each leg still runs the test executables concurrently.
+      - name: Test (threaded + BERTINI_NUM_THREADS=1)
         working-directory: build
         env:
           LD_LIBRARY_PATH: /tmp/boost-base/lib:/tmp/boost-base/lib64
           DYLD_LIBRARY_PATH: /tmp/boost-base/lib
         run: |
           set -e
           jobs=$(nproc 2>/dev/null || sysctl -n hw.ncpu)
-          echo "::group::ctest with OMP_NUM_THREADS unset (per-solve threading on)"
-          env -u OMP_NUM_THREADS ctest --output-on-failure --timeout 600 --parallel "$jobs"
+          echo "::group::ctest with BERTINI_NUM_THREADS unset (per-solve threading on)"
+          env -u BERTINI_NUM_THREADS ctest --output-on-failure --timeout 600 --parallel "$jobs"
           echo "::endgroup::"
-          echo "::group::ctest with OMP_NUM_THREADS=1 (per-solve serial)"
-          OMP_NUM_THREADS=1 ctest --output-on-failure --timeout 600 --parallel "$jobs"
+          echo "::group::ctest with BERTINI_NUM_THREADS=1 (per-solve serial)"
+          BERTINI_NUM_THREADS=1 ctest --output-on-failure --timeout 600 --parallel "$jobs"
           echo "::endgroup::"
 
       # The published wheels ship WITHOUT MPI (we can't know the user's MPI), but this C++ build is
       # MPI-enabled (openmpi is installed above), so we verify the MPI runtime path actually works
       # on each OS we support -- and, crucially, that it gives the SAME answer as the serial solve.
-      # The blackbox CLI reads a file named `input` from the cwd and writes the solution count as
-      # the first line of `main_data`.  mpirun -n 2 = one manager + one worker (the minimal pool).
+      # The blackbox CLI reads a file named `input` from the cwd and writes its results to
+      # `main_data`.  mpirun -n 2 = one manager + one worker (the minimal pool).
       #
       # Run BOTH start-system families, because DistributeSystems broadcasts the start system to the
       # workers and a dropped serialized member is invisible until that exact start system is sent.
       # small.b2 (one affine group) infers total-degree; small_mhom.b2 (two groups) infers the
       # multihomogeneous start system -- whose serialize() was once incomplete, crashing every worker.
-      - name: MPI smoke test (CLI under mpirun)
+      #
+      # With a fixed randomseed, a path's result does not depend on the thread or rank that ran it
+      # (ADR-0071), so main_data must be BYTE-IDENTICAL across: serial, threaded without MPI, and
+      # MPI with one and with two threads per worker -- all set through BERTINI_NUM_THREADS.
+      - name: MPI and thread identity test (CLI)
         working-directory: build/core
         env:
           LD_LIBRARY_PATH: /tmp/boost-base/lib:/tmp/boost-base/lib64
           DYLD_LIBRARY_PATH: /tmp/boost-base/lib
         run: |
           set -e
           for case in small small_mhom; do
-            cp "$GITHUB_WORKSPACE/benchmark/inputs/${case}.b2" input
-            ./bertini2 >/dev/null 2>&1
-            serial=$(head -n1 main_data); rm -f main_data
-            mpirun -n 2 --bind-to none ./bertini2 >/dev/null 2>&1
-            parallel=$(head -n1 main_data); rm -f main_data
-            echo "${case}: solution count serial=$serial  mpi(-n 2)=$parallel"
-            test -n "$serial"
-            test "$serial" = "$parallel"   # MPI must match serial, not merely run
+            { printf 'CONFIG\nrandomseed: 20261005;\n'; tail -n +2 "$GITHUB_WORKSPACE/benchmark/inputs/${case}.b2"; } > input
+            BERTINI_NUM_THREADS=1 ./bertini2 >/dev/null 2>&1; mv main_data serial
+            BERTINI_NUM_THREADS=3 ./bertini2 >/dev/null 2>&1; mv main_data threaded
+            BERTINI_NUM_THREADS=1 mpirun -n 2 --bind-to none ./bertini2 >/dev/null 2>&1; mv main_data mpi_1
+            BERTINI_NUM_THREADS=2 mpirun -n 2 --bind-to none ./bertini2 >/dev/null 2>&1; mv main_data mpi_2
+            echo "${case}: solution count $(head -n1 serial)"
+            test -n "$(head -n1 serial)"
+            for other in threaded mpi_1 mpi_2; do
+              cmp serial "$other"   # identical, not merely the same count
+              echo "  ${other}: identical to serial"
+            done
+            rm -f serial threaded mpi_1 mpi_2
           done
 
   test_cpp_windows:
@@ -360,17 +368,17 @@ jobs:
           ccache --show-stats
 
       # Run under both per-solve thread settings (see the unix C++ test job for the rationale).
-      - name: Test (threaded + OMP_NUM_THREADS=1)
+      - name: Test (threaded + BERTINI_NUM_THREADS=1)
         shell: pwsh
         working-directory: build
         run: |
-          Remove-Item Env:\OMP_NUM_THREADS -ErrorAction SilentlyContinue
-          Write-Host "::group::ctest with OMP_NUM_THREADS unset (per-solve threading on)"
+          Remove-Item Env:\BERTINI_NUM_THREADS -ErrorAction SilentlyContinue
+          Write-Host "::group::ctest with BERTINI_NUM_THREADS unset (per-solve threading on)"
           ctest --output-on-failure --timeout 600 --parallel
           if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
           Write-Host "::endgroup::"
-          $env:OMP_NUM_THREADS = '1'
-          Write-Host "::group::ctest with OMP_NUM_THREADS=1 (per-solve serial)"
+          $env:BERTINI_NUM_THREADS = '1'
+          Write-Host "::group::ctest with BERTINI_NUM_THREADS=1 (per-solve serial)"
           ctest --output-on-failure --timeout 600 --parallel
           if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
           Write-Host "::endgroup::"

CHANGELOG.md

@@ -339,6 +339,30 @@ the identity of the algorithm that answered it (#420).
 
 ### Changed
 
+- **Seed 0 is a seed** (#475, ADR-0072).  `bertini.set_random_seed(0)` and `SetGlobalSeed(0)`
+  meant "draw a seed from entropy", so a script that set seed 0 to make a run reproducible got
+  a different run every time.  0 is now an ordinary seed, as in numpy and Python's `random`.
+  Entropy is asked for by omitting the seed: `set_random_seed()` or `set_random_seed(None)` in
+  Python, `SetGlobalSeedFromEntropy()` in C++.  `set_random_seed` now returns the seed in
+  effect, so the run can be reproduced from it.  The classic input format keeps Bertini 1's
+  meaning: `randomseed: 0`, its default, still draws from entropy in the command-line program.
+  Seed 0 is therefore the one value that does not carry between a Python script and a classic
+  input file.
+- **The thread count is `BERTINI_NUM_THREADS`, and it defaults to every available CPU.**
+  b2 used to read `OMP_NUM_THREADS`, although it uses no OpenMP: its threads are its own
+  `std::thread` pool.  That variable also sets the thread count of numpy's OpenBLAS, so one
+  setting tied the two libraries together, and pinning b2 to one thread for a reproducible run
+  pinned numpy's linear algebra as well.  b2 now reads `BERTINI_NUM_THREADS` and ignores
+  `OMP_NUM_THREADS`.  The order is the variable if set, then the solver's `num_threads`, then
+  the number of CPUs the process may run on, which on Linux respects the affinity mask that
+  `taskset`, cpusets and a launcher's binding set.  That last default is OpenMP's, so threaded
+  runs come without configuration.
+  **MPI ranks change the most.**  A rank used one thread unless `OMP_NUM_THREADS` was set; it
+  now follows the same rule as a standalone solve, `num_threads` included.  How many CPUs a rank
+  may run on depends on how the launcher binds it, so several ranks on one machine can together
+  start more threads than the machine has cores.  Set `BERTINI_NUM_THREADS` whenever you run
+  more than one rank per machine.  A script or job file that set `OMP_NUM_THREADS` for b2
+  should set `BERTINI_NUM_THREADS` instead.
 - **An accuracy estimate names its coordinates** (ADR-0069).  A solution's metadata has two
   accuracy estimates, both the distance between the endgame's last two approximations of the
   root.  `accuracy_estimate` is renamed **`accuracy_estimate_internal_coords`**: it is in the
@@ -460,6 +484,34 @@ the identity of the algorithm that answered it (#420).
 
 ### Fixed
 
+- **A threaded solve gives the same bits as a serial one, and a path no longer depends on the
+  paths before it** (#378, ADR-0071).  Each path's random draws were already deterministic,
+  but state that is not random flowed from one path into the next on a reused tracker or
+  endgame.  Under a thread pool the scheduler chose each thread's predecessors, so a seeded
+  threaded solve differed from run to run in step counts, precision and the last digits of
+  its endpoints, and algorithms built on those endpoints differed with them.  A serial solve
+  depended on path order.  There were five sources:
+  - an adaptive track built its first step at the precision the previous track ended in;
+  - a drop to double left the step size and the time at the old multiple precision;
+  - the condition-number probe was rounded in place at each change of precision;
+  - the endgame's c/k probe was drawn once per endgame object, by whichever path first needed
+    it, or redrawn mid-path after an escalation;
+  - an endgame run in double left an earlier run's multiprecision times in place, so
+    `final_time_used` reported another path's last time.
+  Now a solve at any thread count, under MPI or not, is bit-identical path by path to a
+  serial solve with the same seed: every endpoint and every metadata field but the
+  wall-clock time.  A tracker or endgame driven by hand gives a track the same result as a
+  fresh one would.  Results differ from 4.0.0.dev1 in the last bits and, occasionally, in step
+  counts.
+- **`max_precision_used` is the highest precision a path used, over all of its tracking**
+  (ADR-0071).  The endgame tracks a path in many calls, and the solver's record started afresh
+  with every one, so a precision raised in an earlier endgame sub-track and lowered before the
+  last went unreported: seeded cyclic-5 had paths that reported 30 digits after tracking at 40.
+  The tracker's `FirstPrecisionRecorder` and `MinMaxPrecisionRecorder` now record each track
+  from its very start, the start-point refinement included, and keep recording every track
+  once attached.  `FirstPrecisionRecorder` used to unsubscribe at its first increase and
+  report that track ever after, and a track whose initialization failed was reported with the
+  previous track's values.  A new `PathPrecisionRecorder` records a whole path.
 - **A copied or unpickled system combines with its original** (ADR-0068).  `copy.deepcopy`,
   `copy.copy` and `pickle` go through an archive, and loading from one built variables
   outside the variable factory: a copy was over fresh variables with the same names, and