feat(python/sedonadb): allow 1-D rasters in Raster.from_numpy/lazy - #1097
feat(python/sedonadb): allow 1-D rasters in Raster.from_numpy/lazy#1097james-willis wants to merge 3 commits into
Conversation
Raster.from_numpy() and Raster.lazy() rejected any array/shape with fewer than 2 dimensions, on the assumption the trailing two axes are always a spatial (y, x) pair. That's a Python-layer constraint only -- the underlying RasterSchema already allows empty spatial_dims/ spatial_shape, and Rust's own RasterMetadata::width()/height() are already Option-returning (rust/sedona-raster/src/builder.rs even has a test asserting a single-dim spatial_dims=["x"] case). This blocks representing any purely non-spatial N-D chunk (a plain time series, an embedding vector, a reduction result that collapses below 2-D) as a raster at all. - Relax the from_numpy()/lazy() guard from ">= 2 dims" to ">= 1 dim". - _build_raster: for a 1-D array, spatial_dims/spatial_shape are empty rather than assuming a spatial pair; crs/transform are rejected explicitly in that case (ambiguous without a spatial pair) rather than silently ignored. - Raster.width/height become Optional[int], None for a 1-D raster, matching the Rust side's existing Option. __repr__ falls back to showing the band's source_shape when there's no width/height. - _resolve_dim_names needed no changes: it already requires explicit dim_names for any ndim != 2, which correctly covers ndim == 1 too. Updated the two existing invalid-shape tests (they asserted the old "at least two" rejection for a 1-D shape, which is now valid input) and added test_raster_lazy_1d / test_raster_from_numpy_1d plus cases for the zero-dim and crs/transform-on-1D rejections.
paleolimbot
left a comment
There was a problem hiding this comment.
Ok by me, but are these just "Band"s without the spatial wrapping? Or do we need to update some documentation somewhere that a missing height/width can happen?
|
@paleolimbot Theres some argument to be made here about layers a tensor type and putting spatial raster on top of it but im avoiding that for now. I will update this PR to update documentation for those UDFs. Ill also check to see if the UDFs need any kind of handling |
…er with no spatial (y, x) pair instead of panicking RasterRef::metadata() now returns Result instead of panicking when the raster has no spatial (y, x) pair (empty spatial_shape) or a malformed transform. Per-row UDF closures guard on the error and emit NULL for that row; helpers, dataset builders, and infrastructure propagate the error with `?`; the Display impl renders the available dimension info instead of the width x height / geotransform extent.
|
You could add a |
|
I see your concern. We need to allow 1D bands in the type if we want to support aggregations taht return nonspatial results. Queries that aggregate along 1 or more of the spatial dimensions for example:
This 1D case can of course be represented as some kind of array, but what about when theres 2 nonspatial dimensions?
|
|
Im going to want 0d rasters soon too... |
|
The way the raster type is set up is a very bad fit for returning anything other than 2D things with spatial significance. The "band" as we currently define it is a pretty reasonable way to do this (it's basically just the ndarray with no spatial tag). Can pure ndarray manipulations operate on that data type? They could also accept rasters (they would just not return them because the spatial dimensions would be aggregated away). There are also some aggregations that make sense to return a Raster (very specifically, ones that aggregate into the existing y, x dimensions), but that can't be known at plan time. |
# Conflicts: # rust/sedona-raster-functions/src/executor.rs # rust/sedona-raster-functions/src/footprint.rs # rust/sedona-raster-functions/src/rs_envelope.rs # rust/sedona-raster-functions/src/rs_georeference.rs # rust/sedona-raster-functions/src/rs_geotransform.rs # rust/sedona-raster-functions/src/rs_pixel_functions.rs # rust/sedona-raster-functions/src/rs_setsrid.rs # rust/sedona-raster-functions/src/rs_size.rs # rust/sedona-raster-functions/src/rs_value.rs # rust/sedona-raster-functions/src/rs_values.rs # rust/sedona-raster-gdal/src/gdal_common.rs # rust/sedona-raster-gdal/src/gdal_dataset_provider.rs # rust/sedona-raster-gdal/src/rs_as_geotiff.rs # rust/sedona-raster-gdal/src/rs_as_raster.rs # rust/sedona-raster-gdal/src/rs_clip.rs # rust/sedona-raster-gdal/src/rs_frompath.rs # rust/sedona-raster-gdal/src/rs_metadata.rs # rust/sedona-raster-gdal/src/rs_reproject_match.rs # rust/sedona-raster-gdal/src/rs_resample.rs # rust/sedona-raster-gdal/src/rs_tile.rs # rust/sedona-raster-gdal/src/utils.rs # rust/sedona-raster/src/affine_transformation.rs # rust/sedona-raster/src/array.rs # rust/sedona-raster/src/builder.rs # rust/sedona-raster/src/display.rs # rust/sedona-raster/src/traits.rs # rust/sedona-spatial-join-raster/tests/spatial_join_integration.rs # rust/sedona-testing/src/benchmark_util.rs # rust/sedona-testing/src/raster_spec.rs # rust/sedona-testing/src/rasters.rs
Summary
Raster.from_numpy()andRaster.lazy()rejected any array/shape with fewer than 2 dimensions, on the assumption the trailing two axes are always a spatial(y, x)pair. That's a Python-layer constraint only — the underlyingRasterSchemaalready allows emptyspatial_dims/spatial_shape, and Rust's ownRasterMetadata::width()/height()are alreadyOption-returning (rust/sedona-raster/src/builder.rseven has a test asserting a single-dimspatial_dims=["x"]case).This blocks representing any purely non-spatial N-D chunk (a plain time series, an embedding vector, a reduction result that collapses below 2-D) as a raster at all — hit directly while building an xarray
ChunkManagerEntrypointbackend on top ofsedonadb(.mean(["y", "x"])on a(time, y, x)array produces a(time,)result with nowhere to go).from_numpy()/lazy()guard from ">= 2 dims" to ">= 1 dim"._build_raster: for a 1-D array,spatial_dims/spatial_shapeare empty rather than assuming a spatial pair;crs/transformare rejected explicitly in that case (ambiguous without a spatial pair) rather than silently ignored.Raster.width/heightbecomeOptional[int],Nonefor a 1-D raster, matching the Rust side's existingOption.__repr__falls back to showing the band'ssource_shapewhen there's no width/height._resolve_dim_namesneeded no changes: it already requires explicitdim_namesfor anyndim != 2, which correctly coversndim == 1too.Test plan
pytest python/sedonadb/tests/test_raster.py— 33 passed (updated the two existing invalid-shape tests, which asserted the old "at least two" rejection for a 1-D shape that's now valid input; addedtest_raster_lazy_1d/test_raster_from_numpy_1dplus cases for the zero-dim and crs/transform-on-1D rejections)pytest --doctest-modules python/sedonadb/python/sedonadb/raster.py— 2 passed (new 1-D examples inlazy/from_numpydocstrings)ruff check/ruff format --diffcleantest_rs_value_pointfailures (reproduce identically onorigin/main)