Repository navigation
Conversation
Reading Space Ranger's segmented outputs needs spatialdata-io >= 0.7, which brings spatialdata 0.8, anndata 0.13 and a newer pandas. Move all local modules to one image built on that stack, and fix what breaks under it: - ADATA_EXTEND: anndata 0.13 reports an empty `layers` group as a single `None` key, so count only real keys. - update_table: newer pandas rejects element-wise assignment of mixed values into a float64 column. Assign the whole column with rename/reindex. - nf-test: nft-anndata 0.4.1 cannot read categoricals written by anndata 0.13; update to 0.5.2.
Several per-sample steps carry process_single (6 GB), sized for classic Visium. On Visium HD bins or segmented cells they are killed on the first attempt. Give them more memory through a config selector, alongside the existing one for the merged pre-processing steps. These steps are retried for memory rather than time, so cap their time limit too: escalating to 16 h makes retries hard to schedule.
- nf-core cellpose and stardist modules. The stardist module calls the StarDist API instead of stardist-predict2d, whose --n_tiles takes two values and so cannot tile RGB (YXC) images; untiled, a full-size H&E is killed for memory. Tile counts are derived from the image size. - IMAGE_DOWNSCALE: optional downscaling of the tissue image. - MASK_TO_GEOJSON: label mask to GeoJSON polygons in full-resolution pixels, classified as nucleus or cell. - SDATA_BIN_TO_CELL: group 2 um bins into cells with bin2cell, for processed Space Ranger output. - SEGMENTATION subworkflow tying the image steps together.
New --segmentation_method (spaceranger, cellpose, stardist) and a samplesheet `segmentation_file` column. When either is given, cells replace bins as the analysis unit; the bin table is kept in the SpatialData object. - Raw data: cellpose/stardist segment the tissue image and the result is passed to Space Ranger with --custom-segmentation-file. Whole-cell segmentations (cellpose) are not expanded. - Processed data: Space Ranger's own segmentation is read, or bins are grouped into cells with bin2cell. - Space Ranger's nucleus segmentation is turned off for Visium HD unless cells are requested. - Segmentation runs at full image resolution by default; test profiles downscale, since their tissue image is full size. - Input validation for segmentation on non-HD or incomplete output, and a clear error when no cell table or no cells are produced. - Spatial neighbours use generic coordinates for cells.
Regenerated with nf-test 0.9.4 and Nextflow 25.10.4, as used in CI. Most changes are tool versions from the updated image. The Visium HD pipeline test no longer lists Space Ranger's segmented_outputs, which is now only produced when cells are requested. New snapshots for IMAGE_DOWNSCALE, MASK_TO_GEOJSON and SDATA_BIN_TO_CELL.
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 4.1.0. For more documentation on how to update your pipeline, please see the Synchronisation documentation. |
|
cavenel
marked this pull request as draft
November 9, 2026 11:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds optional cell segmentation for Visium HD, closing #145. When a segmentation is given, cells replace bins as the analysis unit. The bin table is kept in the SpatialData object as
<sample>_bin_table.Cells can come from:
--segmentation_method spaceranger).image(--segmentation_method cellpose|stardist).segmentation_file: GeoJSON or a label mask, in full-resolution pixels.How the segmentation is used depends on the input:
--custom-segmentation-file. Cellpose and StarDist masks are converted to GeoJSON first. Space Ranger then writessegmented_outputs, which the pipeline reads.spaceranger_dir): Space Ranger'ssegmented_outputsare read, or the 2 µm bins are grouped into cells with bin2cell.Other changes this needed:
ADATA_EXTENDlayer count, theSDATA_UPDATE_TABLEreindex and nft-anndata 0.5.2.segmented_outputs/is no longer produced by default. This is a change in behaviour.SDATA_READ_VISIUMandSQUIDPY_SPATIAL_NEIGHBORSnow take a per-sample value in their input tuple (read_cells,coord_type) instead of reading it frommeta. Cells always use generic coordinates.Two nf-core modules are patched. I will open PRs on nf-core/modules for both and link them here:
stardistcalls the StarDist API instead ofstardist-predict2d. The CLI's--n_tilestakes two values, so it can't tile RGB (H&E) images, and an untiled full-size image runs out of memory.ext.argsis passed on topredict_instances.spaceranger/countgets an optional segmentation input.The new modules only have stub tests, because the Visium HD test data is too sparse to segment. Every route was run on the full-size Visium HD Human Lung Cancer dataset except Cellpose, which needs a GPU at that size. Route-level pipeline tests will come with the full-size test profiles in #150.
The PR is large. The commits are kept separate so it can be reviewed one commit at a time.
PR checklist
nf-core pipelines lint).nextflow run . -profile test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).