Skip to content

Cube read path: date-only end bound drops the last day, no #51 paging guards, Jan 2022 cover bias, miss/hit drift, temp NetCDFs #83

Description

@NewGraphEnvironment

Problem

The cube read path (stac_cube_items(), extracted from dft_stac_cube() in #79 and now shared with dft_stac_composite()) has three defects that predate #79. Each one changes which scenes reach a cube, silently.

  1. A date-only datetime end bound drops the last day's scenes. Planetary Computer reads "…/2021-08-27" as 00:00Z, and Sentinel-2 over BC is acquired around 19:00Z. Measured 2026-09-28 on the packaged AOI:

    • 2021-07-01/2021-08-31 gave 23 items, the last on 2021-08-27;
    • 2021-07-01/2021-08-27 gave 22;
    • 2021-07-01T00:00:00Z/2021-08-27T23:59:59Z gave 23.

    dft_stac_composite() already queries with explicit times. dft_stac_cube() passes the user's string through.

  2. The cube paging has none of dft_stac_fetch() reads one STAC page — a wide AOI can build a raster from a truncated item set #51's safeguards. stac_items_paged() in dft_stac_fetch.R checks numberMatched, aborts on duplicate or missing ids, and strips a stale next link. stac_cube_items() calls rstac::items_fetch() bare, so a truncated or duplicated page set reaches the cube unnoticed.

  3. The January 2022 layer of a monthly cube is not a median. Items are split at the 2022-01-25 offset boundary, and the two sides are coalesced with terra::cover(pre, post). The pre-boundary median therefore wins wherever it has data, and post-boundary scenes only fill gaps. dft_stac_composite() refuses such a window. The cube accepts it without comment.

Not verified, but worth checking at the same time: whether gdalcubes reads the SCL mask band with the view's resampling = "bilinear". If it does, mixed-class pixels at cloud edges stop matching a mask value and leak through.

Two more, measured in drift#79's code review (terra 1.9.50, gdalcubes 0.7.5):

  1. A cube's first call and its cache hits disagree. On a miss, dft_stac_cube() returns the in-memory double stack, while the cache holds Float32. A later hit therefore differs by up to 2.98e-08. dft_stac_composite() returns the re-read cache file for exactly this reason.
  2. Assembly leaves large temp files behind. Each build_stack() call writes an uncompressed float64 NetCDF to tempfile() and never unlinks it before the session ends. A floodplain-wide read over a BULK-sized bbox is on the order of 4 GB of tempdir per year.

Why a separate issue

Fixes 1 and 2 change what an existing cache key holds. The key hashes the datetime string, not the item set, so a fix alone leaves stale entries served under unchanged keys. Each fix needs a deliberate cache decision, such as a scheme bump or a key change (code-check.md, "Written data outlives the fix"). That belongs in its own PR, not inside a feature.

Acceptance

  • stac_cube_items() normalizes date-only bounds to full-day times, and cube keys change deliberately with it.
  • stac_cube_items() uses the same paging guards as stac_items_paged(): one implementation, not a second copy.
  • The monthly cube either warns on a straddling month or computes that month exactly. Whichever is chosen is documented in inst/notes/gdalcubes-pc-gotchas.md.

Relates: drift#79, drift#51

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions