Skip to content

HLS (Harmonized Landsat Sentinel-2) as a cube source: Earthdata auth, CMR-STAC query, bit-mask Fmask #82

Description

@NewGraphEnvironment

Problem

Sentinel-2 L2A starts in 2017 for drift's purposes and revisits every ~5 days. That rules out a pre-2017 baseline for dated reference imagery, and it leaves thin seasonal coverage within a year (floodplains#93, drift#79). Harmonized Landsat Sentinel-2 (HLS v2.0, NASA LP DAAC) is 30 m from 2013 with a 2–3 day combined revisit. That fills both gaps.

drift#79 added dft_stac_composite() and the shared cube read path for Sentinel-2 on Planetary Computer, and left HLS to a spike. This issue records what the spike established, measured live on 2026-09-28 against the packaged Neexdzii Kwa reach for July 2023.

What the spike established

Search works without auth, but not with drift's current query shape.

  • The endpoint is https://cmr.earthdata.nasa.gov/stac/LPCLOUD, with collections HLSS30_2.0 (Sentinel-2 input, from 2015-11-28) and HLSL30_2.0 (Landsat input). The reach returned 11 S30 and 7 L30 items, tile T09UXA.
  • intersects returns HTTP 500 on POST. bbox works on GET and POST. An HLS source needs a bbox query, plus the client-side AOI clip drift already does.
  • The CQL2 eo:cloud_cover <= 20 filter is silently ignored: covers 3–100 came back. The cloud pre-filter has to run client-side on properties$eo:cloud_cover.

Reading needs a real Earthdata Login account.

  • Assets live under data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/. An unauthenticated /vsicurl/ read returns 404, not 401, so "file not found" is the wrong diagnosis.
  • earthdatalogin's built-in default credentials no longer work. edl_set_token() returns HTTP 401. The netrc path reads the login page ("not recognized as being in a supported file format").
  • Auth path to implement and verify: earthdatalogin::edl_netrc(username, password), with credentials from EARTHDATA_USER / EARTHDATA_PASSWORD. This sets GDAL_HTTP_NETRC_FILE and the cookie jar for GDAL, and it replaces Planetary Computer's sign_fn.
  • Not yet verified end to end, because the spike had no account. Verifying that one /vsicurl/ COG read and one gdalcubes cube succeed is the first task here.

Band roles differ between the two collections.

role S30 L30
blue / green / red B02 / B03 / B04 B02 / B03 / B04
nir B8A (narrow NIR, the harmonized one) B05
swir16 B11 B06
swir22 B12 B07

A single "hls" source needs per-collection role maps, or it becomes two sources (hls-s30, hls-l30) composited together.

The mask is a bit mask.

  • Fmask bits: 1 cloud, 2 adjacent cloud, 3 cloud shadow, 4 snow/ice, 5 water; 6–7 aerosol. This is not SCL-style class values.
  • gdalcubes::image_mask() takes bits =, so the source config needs a mask-bits field beside mask_values, and stac_cube_assemble() must pass it through.
  • Reflectance scale is 1e-4 with no offset, so there is no baseline split.

Scope

  1. An hls cube source (or two) in dft_stac_config(), with the roles above and bit-mask config.
  2. An auth path distinct from sign_fn: an Earthdata Login netrc or token, set for the read and restored after it, the way stac_cube_session() restores GDAL config.
  3. A query adapter for CMR-STAC: a bbox query, a client-side cloud filter, and pagination checked against CMR's next links.
  4. dft_stac_composite() and dft_stac_cube() accept the new source unchanged.

Acceptance

  • One live composite from HLS S30 over the packaged AOI, with credentials from env vars. The e2e skips cleanly without them.
  • Offline tests for the role maps, the bit-mask config and the client-side cloud filter.
  • A 2015 or 2016 composite exists for a floodplains#93 window, which Sentinel-2 L2A cannot provide.

Relates: drift#79, floodplains#93

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions