n4_biasfield memory grows with image size instead of streaming #215

n4_biasfield applies a precomputed low-resolution N4 bias field to the image at the segmentation level. It should stream block by block, but its peak memory grows with the size of the image. Right now it is the largest memory request in the workflow, and it fails on the first attempt on some subjects.

Measurements

All runs used 128 threads, on .ims inputs from trident_rescans (benchmarks from #213 and #214):

• On the first attempt of the last row (147 GB requested), dask workers kept dying (KilledWorker on _fused_block tasks). It succeeded on the retry with 294 GB. CPU efficiency was about 5%.
• The current request, scaled_mem_mb(128000, 500), is fitted to these numbers: a large fixed base plus about 0.5× the image size.

Where the memory might go

These are untested candidates, in spimquant/workflow/scripts/n4_biasfield.py and ZarrNii 0.21.6:

1. Round trip through a temporary zarr for .ims: znimg_in.to_ome_zarr(temp_dir) uses tempfile.TemporaryDirectory(), which is /tmp by default. If /tmp is RAM-backed on the compute nodes, the whole image ends up counted against the job's memory.
2. Writing the pyramid: to_ome_zarr(..., match_scale_factors_from=...) writes level 0 plus downsampled levels. If the levels are computed from one shared graph, dask may keep level-0 blocks in memory until every level is written.
3. Per-block interpolation in ZarrNii._apply_scaled_processing_map_blocks: scipy.ndimage.map_coordinates builds a coordinate grid of 3 × voxels in float64 for each block, about 24 bytes per voxel. It does this twice when a mask is given. For a 512³ block that is about 3 GB per call. With threads_per_worker=16 for .ims, a worker runs 16 blocks at once, about 48 GB, while memory_limit="auto" gives each of the 8 workers 18 GB of a 147 GB job. The scaling is axis-aligned, so separable 1-D linear interpolation would need only a few 1-D weight vectors per axis.
4. dask settings: threads_per_worker (16 for .ims, 2 otherwise) and the chunk sizes on read (256³) and write (target_chunk_size=512).

Plan

1. Reproduce on one subject at level 1, then measure the steps one at a time: read only, then apply the bias field without writing, then write only level 0, then write the full pyramid. Use --benchmark and the dask dashboard or a memory profile. Check whether /tmp is tmpfs on the nodes.
2. Fix in ZarrNii whatever turns out to be ZarrNii's (likely items 2 and/or 3): separable interpolation in map_blocks, and a pyramid write that streams. Open a ZarrNii issue or PR once the cause is known.
3. Fix in SPIMquant whatever belongs to the script: write the .ims temp zarr to the job's disk scratch (resources.tmpdir) or skip it, and tune threads_per_worker and the chunk sizes.
4. Refit n4_biasfield resources once memory follows threads × chunk size. Run tools/summarize_benchmarks.py on runs at levels 0 and 1, and compare with what Slurm measures (--slurm-efficiency-report), since Slurm reports more than psutil.
5. Check the other dask rules for the same pattern. When the request is too small, workers pause at their per-worker limit, so the benchmarked peak stays low and hides it. compute_filtered_regionprops was raised for this reason in Group the short jobs of each subject into one cluster job per stage #214. signed_distance_transform also had workers die on its first attempt.

Done when

• Peak memory no longer grows with image size at a fixed chunk size and thread count.
• n4_biasfield succeeds on the first attempt at levels 0 and 1 with a request well under the current one.