Add spatial-temporal metadata embeddings to CropModel by bw4sz · Pull Request #1334 · weecology/DeepForest

bw4sz · 2026-03-05T18:52:52Z

Summary

Adds optional late-fusion spatial-temporal metadata (lat, lon, date) to CropModel for improved species classification
Sinusoidal encoding of coordinates + day-of-year through a small MLP (~1.5% of image feature vector), concatenated with ResNet-50 features before the classifier
Fully backward compatible — metadata is optional at both training and inference time; model gracefully degrades to image-only predictions when metadata is absent

Key changes

SpatialTemporalEncoder: Sinusoidal positional encoding + MLP (6 → configurable dim, default 32)
CropModel: Splits into backbone + metadata encoder + classifier when use_metadata=True
MetadataImageFolder: Wraps ImageFolder with a CSV sidecar (filename,lat,lon,date) for training
BoundingBoxDataset: Supports per-crop metadata dict for inference
predict_tile: Accepts metadata={"lat": ..., "lon": ..., "date": "YYYY-MM-DD"}
Config: use_metadata, metadata_dim, metadata_dropout
19 new tests covering encoder, model, datasets, checkpoint save/load, and full training integration
Documentation added to CropModel user guide and configuration reference

Test plan

All 19 new metadata tests pass (pytest tests/test_metadata_cropmodel.py)
All 8 existing crop model tests pass (pytest tests/test_crop_model.py)
Full test suite passes on CI
Manual smoke test with real field data

🤖 Generated with Claude Code

Note

Medium Risk
Touches core CropModel training/inference data flow and changes batch/forward signatures to support metadata; defaults remain unchanged, but incorrect metadata wiring or shape handling could break crop-model prediction/training in edge cases.

Overview
Adds optional spatial-temporal metadata (lat/lon/date) support to CropModel, including a new SpatialTemporalEncoder and a backbone+metadata+classifier path when cropmodel.use_metadata is enabled, while keeping image-only behavior as the default.

Training can now load per-image metadata from a sidecar CSV via CropModel.load_from_disk(..., metadata_csv=...), and inference can pass image-level metadata through deepforest.predict_tile(..., metadata=...) which is expanded to per-crop tensors and threaded through predict/BoundingBoxDataset.

Updates config/schema/docs to expose use_metadata, metadata_dim, and metadata_dropout, and adds a dedicated test suite covering encoder behavior, training/predict steps, dataset wrappers, and checkpoint save/load.

^{Written by Cursor Bugbot for commit c763c17. This will update automatically on new commits. Configure here.}

Species distributions vary by location and season. This adds an optional late-fusion pathway that encodes (lat, lon, date) as a small sinusoidal embedding (~1.5% of image features) concatenated with ResNet-50 features before the classifier. Metadata is fully optional — the model gracefully degrades to image-only predictions when metadata is absent. Key changes: - SpatialTemporalEncoder with sinusoidal encoding + small MLP - CropModel.create_model splits into backbone + classifier when use_metadata=True - MetadataImageFolder wraps ImageFolder with CSV sidecar for training - BoundingBoxDataset supports per-crop metadata for inference - predict_tile accepts metadata dict (lat, lon, date) - Config: use_metadata, metadata_dim, metadata_dropout - 19 new tests covering encoder, model, datasets, and integration - Documentation for CropModel docs and configuration reference Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

bw4sz · 2026-03-08T16:58:32Z

I changed something small, I don't agree with bugbot.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Reapply the CropModel metadata changes on current main so the PR branch can merge cleanly.

Co-authored-by: Cursor <cursoragent@cursor.com>

bw4sz · 2026-05-18T19:43:14Z

@jveitchmichaelis can you skim this. There was a lot of cursor generated code at the beginning and I'm starting to take ownership over it as it seems to be effective and well balanced when I tested it in the BOEM repo.

There are implementation pieces I'm not sure about, and then there are scientific things that merit further review. Like what happens if we see some that is not supposed to occur in a new case, I want to manually change a few test items to see if the metadata overrides the vision model. It's a small portion of the concat features but I will test it. In terms of engineering and our workflows, I am playing with this process of working with the agents to prototype something, check it out on a different project and then assume ownership of the PR for DeepForest. That hand-off is probably awkward. I want to see which things strike you as needing developer time. @ethanwhite just keeping you abreast as a trial pattern. I'm taking it out of WIP, but not triggering review yet?

Co-authored-by: Cursor <cursoragent@cursor.com>

bw4sz · 2026-06-01T18:34:31Z

+            else:
+                meta_dim = self.classifier.in_features - image_features.shape[1]
+                meta_features = torch.zeros(
+                    x.shape[0], meta_dim, device=x.device, dtype=x.dtype


@jveitchmichaelis let's discuss the use of x.device here, flagging this as something that worries me with #1391

The device move here could be to the device that self.backbone is on, not to x (although if x is incorrect, we would fail on L519 anyway).

But this comes through predict_step which should handle that? So mirroring .x should be fine.

updated to self.backbone.device

bw4sz · 2026-06-01T18:39:42Z

Okay, I have taken control over this branch. I am happy with the commits with the exception of the two comments i've flagged. I want to make sure this makes sense if we every want a transformer backbone to the crop classification model (not detection) and whether we learned anything about pytorch lightning handles the .device internally within single/multi-gpu. Other than that, testing on the BOEM pipeline shows this works nicely and I think users will appreciate this as a first step towards greater ecological realism. Getting this merged will allows us to begin to experiment with pretraining and the grain and intensity of the metadata prior.

…ropModel docs and reduced unneeded tests Co-authored-by: Cursor <cursoragent@cursor.com>

jveitchmichaelis

I think this looks OK. There are a couple of style nits that would make the train/predict steps a bit cleaner, but nothing I would necessarily/urgently change here.

If you're happy with the device placement then I think this is good to go.

jveitchmichaelis · 2026-06-04T20:46:53Z

+            else:
+                meta_dim = self.classifier.in_features - image_features.shape[1]
+                meta_features = torch.zeros(
+                    x.shape[0], meta_dim, device=x.device, dtype=x.dtype


The device move here could be to the device that self.backbone is on, not to x (although if x is incorrect, we would fail on L519 anyway).

But this comes through predict_step which should handle that? So mirroring .x should be fine.

jveitchmichaelis · 2026-06-04T20:49:18Z

-        x, y = batch
-        outputs = self.forward(x)
+        if len(batch) == 3:
+            x, y, metadata = batch


In future we could probably have this more cleanly as batch.x, batch.y, batch.metadata and have the collation handle it. As opposed to a conditional here and having to set metadata anyway.

Let's do this in a separate PR if needed, since it's not related to the behavior of the current collate.

bw4sz · 2026-07-07T18:45:57Z

Picking this back up, once it passes, @jveitchmichaelis merge if you are ready.

jveitchmichaelis

I had a re-read:

Some comments about docs where things could be expanded upon / also docstrings
A few comments about the architecture of the changes. The modeling looks fine, but it's getting a bit messy with conditions everywhere. I would just make a separate subclass or a wrapper as we do with the dataset. Let's look at that later.
Needs some tests for the dataset + check that embeddings are computed in range etc?

jveitchmichaelis · 2026-07-14T21:24:46Z

+crop_model.trainer.validate(crop_model)
+
+# Save checkpoint
+crop_model.trainer.save_checkpoint("model.ckpt")


Do we have an example for loading? The difference between lightning checkpoint, huggingface model + config in a folder, and a plain torch checkpoint can be confusing for users.

jveitchmichaelis · 2026-07-14T21:25:45Z

+
+When `use_metadata=True`, the CropModel:
+
+1. Encodes `(lat, lon, day_of_year)`


Add a bit more detail to how the encoding works (e.g. position/rotation)

jveitchmichaelis · 2026-07-14T21:26:38Z

+result = m.predict_tile(
+    path="image.tif",
+    crop_model=crop_model,
+    metadata={"lat": 35.2, "lon": -120.4, "date": "2024-06-15"}


Should probably specify somewhere what requirements are for this - are all necessary, only some?

jveitchmichaelis · 2026-07-14T21:27:01Z


-A `CropModel` is a PyTorch Lightning object and can also be used like any other model.
+- `filename` matches the image basename
+- `date` is an ISO format string


What format/convetion lat/lon?

jveitchmichaelis · 2026-07-14T21:27:46Z


-### Sampler
+The map below shows a relative metadata prior for Northern Gannet
+(`Morus bassanus`) on April 15, 2024. It reflects the learned metadata branch,


Can we remove the LLM prose here please (also unclear to me what "It reflects the learned metadata branch, not image evidence" actually means)

jveitchmichaelis · 2026-07-14T22:32:28Z

+class MetadataImageFolder(Dataset):
+    """Wrapper that adds spatial-temporal metadata to an ImageFolder dataset.
+
+    Expects a CSV file with columns: filename, lat, lon, date.


Same as other comments - need to specify the lat lon format (e.g. decimal degrees). I suppose it doesn't matter as long as the column is consistent.

jveitchmichaelis · 2026-07-14T22:33:46Z

+    """Wrapper that adds spatial-temporal metadata to an ImageFolder dataset.
+
+    Expects a CSV file with columns: filename, lat, lon, date.
+    The date column should be an ISO format string (e.g., "2024-06-15")


Would be better to use fromisoformat which is more flexible: https://docs.python.org/3/library/datetime.html#datetime.date.fromisoformat

jveitchmichaelis · 2026-07-14T22:36:30Z

+        filepath = self.image_folder.samples[idx][0]
+        filename = os.path.basename(filepath)
+
+        if filename in self._meta_lookup:


Should the fallback be separate for each field? e.g. if you have lat/lon but no date.

I suggest storing the metadata (per file) as a NamedTuple which is also a bit more defensive against getting the order wrong.

jveitchmichaelis · 2026-07-14T22:54:11Z

-        )
+        use_metadata = self.config["cropmodel"].get("use_metadata", False)
+
+        if use_metadata:


It's probably transparent to the user, but feels awkward to have backbone/model/classifier and then have metadata models not have model defined and vice versa. I am OK with this for now, but I would consider sub-classing the CropModel to avoid having conditional statements everywhere.

jveitchmichaelis · 2026-07-14T22:56:14Z

@@ -0,0 +1,76 @@
+"""Tests for spatial-temporal metadata embeddings in CropModel."""


Probably need a few tests that loading data with metadata columns are parsed correctly + we get sensible embeddings out (generally, add tests for the modified dataset class)

jveitchmichaelis · 2026-07-14T22:59:44Z

(also needs squash)

bw4sz added Feature Request New feature or request in progress Someone is currently working on this. Check with the assigned user before working on it. labels Mar 5, 2026

bw4sz added 3 commits March 5, 2026 15:31

simplify text block

5a377c5

doc error

a3cdf14

improve docs

0a9f598

cursor Bot reviewed Mar 7, 2026

View reviewed changes

Comment thread src/deepforest/model.py Outdated

simplify predict step

1c45062

inference batch cannot be length 3

0c57f8a

cursor Bot reviewed Mar 9, 2026

View reviewed changes

Comment thread src/deepforest/model.py

bw4sz and others added 3 commits March 10, 2026 10:28

doc error

c763c17

Merge main into PR 1334

b37826f

Reapply the CropModel metadata changes on current main so the PR branch can merge cleanly.

Add metadata prior visualization docs

489879a

Co-authored-by: Cursor <cursoragent@cursor.com>

bw4sz changed the title ~~WIP: Add spatial-temporal metadata embeddings to CropModel~~ Add spatial-temporal metadata embeddings to CropModel May 18, 2026

Trim metadata CropModel tests to essential coverage

8f7c9a4

Co-authored-by: Cursor <cursoragent@cursor.com>

bw4sz commented Jun 1, 2026

View reviewed changes

Comment thread src/deepforest/model.py

bw4sz commented Jun 1, 2026

View reviewed changes

bw4sz requested a review from jveitchmichaelis June 1, 2026 18:37

I refined the initial cursor prediction to make it leaner, improved C…

919c84d

…ropModel docs and reduced unneeded tests Co-authored-by: Cursor <cursoragent@cursor.com>

bw4sz force-pushed the claude/friendly-beaver branch from 85d8247 to 919c84d Compare June 1, 2026 21:21

jveitchmichaelis reviewed Jun 4, 2026

View reviewed changes

change x.device to self.device

26ff5be

bw4sz force-pushed the claude/friendly-beaver branch from 4285986 to 26ff5be Compare July 7, 2026 20:00

jveitchmichaelis requested changes Jul 14, 2026

View reviewed changes


		When `use_metadata=True`, the CropModel:

		1. Encodes `(lat, lon, day_of_year)`

		@@ -0,0 +1,76 @@
		"""Tests for spatial-temporal metadata embeddings in CropModel."""

Uh oh!

Conversation

bw4sz commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key changes

Test plan

Uh oh!

Uh oh!

bw4sz commented Mar 8, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bw4sz commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jveitchmichaelis Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bw4sz commented Jun 1, 2026

Uh oh!

jveitchmichaelis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jveitchmichaelis Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bw4sz commented Jul 7, 2026

Uh oh!

jveitchmichaelis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jveitchmichaelis commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bw4sz commented Mar 5, 2026 •

edited

Loading

bw4sz commented May 18, 2026 •

edited

Loading

jveitchmichaelis Jun 4, 2026 •

edited

Loading

jveitchmichaelis left a comment •

edited

Loading

jveitchmichaelis Jun 4, 2026 •

edited

Loading

jveitchmichaelis left a comment •

edited

Loading