Cell Segmentation

CHANGELOG.md

@@ -19,6 +19,10 @@ compatible with further downstream analyses and/or exploration in _e.g._
 ### `Added`
 
 - Add pickled slot data outputs and `adata/extend`; don't output "raw" layer in raw data [[#153](https://github.com/nf-core/spatialvi/pull/153/)]
+- Add optional Visium HD cell segmentation with Space Ranger, Cellpose, StarDist or a samplesheet `segmentation_file`, using cells instead of bins for the analysis [[#145](https://github.com/nf-core/spatialvi/issues/145) and [#161](https://github.com/nf-core/spatialvi/pull/161)]
+- Turn off Space Ranger's nucleus segmentation for Visium HD unless cells are requested, so `segmented_outputs/` is no longer produced by default [[#161](https://github.com/nf-core/spatialvi/pull/161)]
+- Move local modules to the SpatialData 0.8 stack (spatialdata-io 0.7, anndata 0.13) and add bin2cell [[#161](https://github.com/nf-core/spatialvi/pull/161)]
+- Increase memory for per-sample steps on Visium HD data, and cap their time limit at 4 h per attempt [[#161](https://github.com/nf-core/spatialvi/pull/161)]
 - Deprecate `quartonotebook` module in favour of `quarto/notebook` [[#148](https://github.com/nf-core/spatialvi/pull/148)]
 - Separate reports into separate scripts [[#137](https://github.com/nf-core/spatialvi/pull/137)]
 - Use topic channels for software versions [[#133](https://github.com/nf-core/spatialvi/pull/133)]

CITATIONS.md

@@ -14,6 +14,16 @@
 
   > Virshup I, Rybakov S, Theis FJ, Angerer P, Wolf FA. bioRxiv 2021.12.16.473007; doi: https://doi.org/10.1101/2021.12.16.473007
 
+- [bin2cell](https://github.com/Teichlab/bin2cell)
+
+  > Polański K, Bartolomé-Casado R, Sarropoulos I, Xu C, England N, Jahnsen FL, Teichmann SA, Yayon N. Bin2cell reconstructs cells from high resolution Visium HD data. Bioinformatics. 2024;40(9):btae546. doi: 10.1093/bioinformatics/btae546
+
+- [Cellpose](https://github.com/MouseLand/cellpose)
+
+  > Stringer C, Wang T, Michaelos M, Pachitariu M. Cellpose: a generalist algorithm for cellular segmentation. Nat Methods 18, 100–106 (2021). doi: 10.1038/s41592-020-01018-x
+
+  > Pachitariu M, Rariden M, Stringer C. Cellpose-SAM: superhuman generalization for cellular segmentation. bioRxiv 2025.04.28.651001; doi: https://doi.org/10.1101/2025.04.28.651001
+
 - [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
 
 > Andrews, S. (2010). FastQC: A Quality Control Tool for High Throughput Sequence Data [Online].
@@ -42,6 +52,12 @@
 
   > Palla G, Spitzer H, Klein M et al. Squidpy: a scalable framework for spatial omics analysis. Nat Methods 19, 171–178 (2022). doi: https://doi.org/10.1038/s41592-021-01358-2
 
+- [StarDist](https://github.com/stardist/stardist)
+
+  > Schmidt U, Weigert M, Broaddus C, Myers G. Cell Detection with Star-Convex Polygons. MICCAI 2018. doi: 10.1007/978-3-030-00934-2_30
+
+  > Weigert M, Schmidt U, Haase R, Sugawara K, Myers G. Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy. WACV 2020. doi: 10.1109/WACV45572.2020.9093435
+
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)

README.md

@@ -31,6 +31,12 @@ the following steps:
     <img title="Spatialvi Workflow" src="docs/images/spatialvi_subway.png" width=100%>
 </p>
 
+For Visium HD data, cells can optionally replace bins as the analysis unit,
+using a cell segmentation from Space Ranger, [Cellpose](https://github.com/MouseLand/cellpose),
+[StarDist](https://github.com/stardist/stardist) or a user-supplied file. For
+data already processed by Space Ranger, the last three are applied with
+[bin2cell](https://github.com/Teichlab/bin2cell).
+
 The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool
 to run tasks across multiple compute infrastructures in a very portable manner.
 It uses Docker/Singularity containers making installation trivial and results

bin/report.qmd

@@ -129,9 +129,18 @@ def display_table(table, hide_index=True, format_dict=None):
 # Load SpatialData
 sdata = spatialdata.read_zarr(input_sdata)
 
-# Get the table (assuming single sample, take first table)
-table_names = list(sdata.tables.keys())
-table_name = table_names[0]
+# Get the main table (assuming single sample; skip the Visium HD bin table,
+# which only exists next to a "<sample>_table" of cells)
+table_name = next(
+    (
+        t for t in sdata.tables
+        if t.endswith("_table")
+        and not (t.endswith("_bin_table") and t[: -len("_bin_table")] + "_table" in sdata.tables)
+    ),
+    None,
+)
+if table_name is None:
+    raise ValueError(f"No main table in SpatialData object; tables present: {list(sdata.tables)}")
 adata = sdata.tables[table_name]
 
 # Extract sample ID from table name

conf/modules.config

@@ -39,14 +39,59 @@ process {
     }
 
     withName: SPACERANGER_COUNT {
-        ext.args = '--create-bam="false"'
+        // cellpose gives whole cells, which are not expanded; HD without cells skips segmentation
+        ext.args = { [
+            '--create-bam="false"',
+            segmentation ? "--nucleus-expansion-distance-micron=${params.segmentation_method == 'cellpose' ? 0 : params.segmentation_expansion_distance}" : '',
+            (meta.hd && !meta.cells) ? '--nucleus-segmentation=false' : ''
+        ].join(' ') }
         publishDir = [
             path: { "${params.outdir}/${meta.id}/spaceranger" },
             mode: params.publish_dir_mode,
             saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
         ]
     }
 
+    // Per-sample steps on Visium HD bins or cells need more than process_single
+    withName: 'SCANPY_FILTER|SCANPY_CALCULATE_QC_METRICS|SDATA_TO_LEGACY_ANNDATA|SDATA_UPDATE_TABLE|.*SPATIAL:SQUIDPY_.*' {
+        memory = { 36.GB * task.attempt }
+    }
+
+    // Retried for memory rather than time; keep retries schedulable
+    withName: 'SCANPY_.*|SDATA_.*|SQUIDPY_.*|IMAGE_DOWNSCALE|MASK_TO_GEOJSON' {
+        time = { 4.h * task.attempt }
+    }
+
+    withName: 'CELLPOSE' {
+        ext.args = '--no_npy'
+        // Cellpose-SAM needs a GPU for Visium HD; it will not finish on CPU
+        cpus   = { 12    * task.attempt }
+        memory = { 72.GB * task.attempt }
+        time   = { 8.h   * task.attempt }
+        // cellpose writes the mask into a per-sample subdirectory
+        publishDir = [
+            path: { "${params.outdir}/${meta.id}/segmentation" },
+            mode: params.publish_dir_mode,
+            pattern: "**/*masks.tif",
+            saveAs: { filename -> filename.tokenize('/').last() }
+        ]
+    }
+
+    withName: 'STARDIST' {
+        // Full-resolution prediction peaks above process_medium
+        cpus   = { 12    * task.attempt }
+        memory = { 72.GB * task.attempt }
+        time   = { 8.h   * task.attempt }
+    }
+
+    withName: 'STARDIST|MASK_TO_GEOJSON' {
+        publishDir = [
+            path: { "${params.outdir}/${meta.id}/segmentation" },
+            mode: params.publish_dir_mode,
+            pattern: "*.{tif,geojson}"
+        ]
+    }
+
     withName: FASTQC {
         publishDir = [
             path: { "${params.outdir}/${meta.id}/fastqc" },
@@ -83,11 +128,11 @@ process {
 
     // The Seqera image used in multiple modules of this pipeline contains many
     // different packages, the build information for which you can find here:
-    // https://wave.seqera.io/view/builds/bd-95f731fde0b9ddef_1.
+    // https://wave.seqera.io/view/builds/bd-d457f4c6ebb7b325_1.
     withName: REPORT {
         ext.prefix = { "${notebook.baseName}-${meta.id}" }
         ext.args = { "--metadata subtitle:'Sample: ${meta.id}'" }
-        container = 'community.wave.seqera.io/library/harmonypy_scanorama_gcc_gxx_pruned:95f731fde0b9ddef'
+        container = 'community.wave.seqera.io/library/bin2cell_harmonypy_scanorama_imagecodecs_pruned:d457f4c6ebb7b325'
         publishDir = [
             path: { "${params.outdir}/${meta.id}" },
             mode: params.publish_dir_mode,
@@ -137,7 +182,7 @@ process {
 
     withName: REPORT_INTEGRATED {
         ext.prefix = { "${notebook.baseName}" }
-        container = 'community.wave.seqera.io/library/harmonypy_scanorama_gcc_gxx_pruned:95f731fde0b9ddef'
+        container = 'community.wave.seqera.io/library/bin2cell_harmonypy_scanorama_imagecodecs_pruned:d457f4c6ebb7b325'
         publishDir = [
             path: { "${params.outdir}/integration" },
             mode: params.publish_dir_mode,

conf/test.config

@@ -23,6 +23,7 @@ params {
     config_profile_description = 'Test pipeline functionality, including Space Ranger v2'
 
     // Input and output
+    segmentation_downscale = 4 // test images are full resolution; keep segmentation fast
     input = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-brain-cancer-11-mm-capture-area-ffpe-2-standard_v2_ffpe_cytassist/samplesheet_spaceranger.csv"
     spaceranger_probeset = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-brain-cancer-11-mm-capture-area-ffpe-2-standard_v2_ffpe_cytassist/outs/probe_set.csv"
     spaceranger_reference = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/homo_sapiens_chr22_reference.tar.gz"

conf/test_downstream.config

@@ -23,6 +23,7 @@ params {
     config_profile_description = 'Test pipeline for downstream (post-Space Ranger) functionality'
 
     // Input and output
+    segmentation_downscale = 4 // test images are full resolution; keep segmentation fast
     input  = 'https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-brain-cancer-11-mm-capture-area-ffpe-2-standard_v2_ffpe_cytassist/samplesheet_downstream.csv'
     spaceranger_probeset = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-brain-cancer-11-mm-capture-area-ffpe-2-standard_v2_ffpe_cytassist/outs/probe_set.csv"
     spaceranger_reference = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/homo_sapiens_chr22_reference.tar.gz"

conf/test_skip_downstream.config

@@ -23,6 +23,7 @@ params {
     config_profile_description = 'Test pipeline with skipping of all downstream steps'
 
     // Input and output
+    segmentation_downscale = 4 // test images are full resolution; keep segmentation fast
     input = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-brain-cancer-11-mm-capture-area-ffpe-2-standard_v2_ffpe_cytassist/samplesheet_spaceranger.csv"
     spaceranger_probeset = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-brain-cancer-11-mm-capture-area-ffpe-2-standard_v2_ffpe_cytassist/outs/probe_set.csv"
     spaceranger_reference = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/homo_sapiens_chr22_reference.tar.gz"

conf/test_spaceranger_hd.config

@@ -23,6 +23,7 @@ params {
     config_profile_description = 'Test pipeline functionality on Visium HD file'
 
     // Input and output
+    segmentation_downscale = 4 // test images are full resolution; keep segmentation fast
     input = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-lung-cancer-post-xenium_hd_ffpe/samplesheet_spaceranger_hd.csv"
     spaceranger_probeset = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-lung-cancer-post-xenium_hd_ffpe/Visium_HD_Human_Lung_Cancer_HD_Only_Experiment2_probe_set.csv"
     spaceranger_reference = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/GRCh38.tar.gz"

conf/test_spaceranger_v1.config

@@ -23,6 +23,7 @@ params {
     config_profile_description = 'Test pipeline functionality, including Space Ranger v1'
 
     // Input and output
+    segmentation_downscale = 4 // test images are full resolution; keep segmentation fast
     input  = 'https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-ovarian-cancer-1-standard_v1_ffpe/samplesheet_spaceranger.csv'
     spaceranger_probeset = 'https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/human-ovarian-cancer-1-standard_v1_ffpe/Visium_Human_Transcriptome_Probe_Set_v1.0_GRCh38-2020-A.csv'
     spaceranger_reference = "https://raw.githubusercontent.com/nf-core/test-datasets/spatialvi/testdata/homo_sapiens_chr22_reference.tar.gz"

docs/output.md

@@ -16,6 +16,7 @@ The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes
 data using the following steps:
 
 - [Space Ranger](#space-ranger)
+- [Cell segmentation](#cell-segmentation)
 - [FastQC](#fastqc)
 - [Per-sample data](#per-sample-data)
 - [Per-sample reports](#per-sample-reports)
@@ -47,6 +48,27 @@ All files produced by Space Ranger are currently published as output of this
 pipeline, regardless if they're being used downstream or not; you can find more
 information about these files at the [10X website](https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/output/overview).
 
+## Cell segmentation
+
+<details markdown="1">
+<summary>Output files</summary>
+
+- `<SAMPLE>/segmentation/`
+  - `*_cp_masks.tif` or `*.stardist.tif`: Label mask from cellpose or stardist,
+    on the (optionally downscaled) image.
+  - `*_segmentation.geojson`: Cell polygons in full-resolution image pixels.
+
+</details>
+
+Only produced with `--segmentation_method cellpose` or `stardist`. Space Ranger
+segmentation results are in `<SAMPLE>/spaceranger/outs/segmented_outputs/`.
+
+When a segmentation is used, cells replace bins as the analysis unit. The
+SpatialData objects (see [Integration](#integration)) then also contain the
+bin table as `<SAMPLE>_bin_table`, and the cell polygons as `<SAMPLE>_cells`
+shapes when bins were grouped with bin2cell (otherwise as read from Space
+Ranger's `segmented_outputs/`).
+
 ## FastQC
 
 <details markdown="1">
@@ -123,6 +145,8 @@ selected per sample and ranked by in how many samples they are highly variable.
   - `merged.zarr`: Merged SpatialData object containing all samples (before
     integration).
   - `<METHOD>.zarr`: Integrated SpatialData object with batch-corrected results.
+    With cell segmentation, both `.zarr` objects also hold each sample's cell
+    shapes and `<SAMPLE>_bin_table` table (see [Cell segmentation](#cell-segmentation)).
   - `<METHOD>.h5ad`: Integrated AnnData object with batch-corrected embeddings
     and clustering. The PCA of the merged data is in `obsm["X_pca_merged"]`,
     while `obsm["X_pca"]` holds each spot's coordinates in the PCA of its own