Skip to content

perf: batch PyLovo linking by cell index, not a grid join - #213

Merged
jravani merged 2 commits into
mainfrom
perf/link-grid-batches
Oct 22, 2026
Merged

jravani merged 2 commits into
mainfrom
perf/link-grid-batches

Conversation

@jravani

@jravani jravani commented Oct 22, 2026

Copy link
Copy Markdown
Member

-link-pylovo spent 29 minutes batching buildings before linking began, on one country database holding Deggendorf and Köln (27,420 buildings). The linking itself took under a second.

Cause

GetGridBatches built an ST_SquareGrid over the envelope of all buildings and joined every building's point to every cell. The two regions span 438 x 233 km, so 103,400 cells; the planner estimated 1,000 and chose a nested loop: about 2.8 billion ST_Intersects calls.

Nested Loop  (cost=3976.53..360242410.42 rows=27420)
  Join Filter: st_intersects(st_pointonsurface(b.geom), (st_squaregrid('1000', ...)).geom)
  ->  CTE Scan on all_buildings b  (rows=27420)
  ->  Materialize  (rows=1000)
        ->  ProjectSet ... st_squaregrid

Change

  • Each building's cell is floor(x / size), floor(y / size) of a point inside its footprint, and batches are grouped by that index. The cells are the same origin-aligned squares, every building still lands in exactly one batch, and empty cells are never built.
  • CHANGELOG.md: Fixes entry.
GroupAggregate (actual rows=49)
  ->  Gather Merge -> Sort -> Parallel Seq Scan on lod2_building (actual rows=9140, loops=3)
Execution Time: 38.983 ms

Tests

  • Same database: 49 batches before and after, each of the 27,420 buildings in exactly one; 29 min before, 39 ms after.
  • The five TestGetGridBatches_* integration tests and the rest of ./internal/process/ pass. No test fails before this change: the output is unchanged and only the cost differs.

GetGridBatches built an ST_SquareGrid over the envelope of all
buildings and joined every building's point to every cell. For one
country database holding two regions far apart the envelope is mostly
empty: Deggendorf and Koeln, 27,420 buildings, span 438 x 233 km, so
103,400 cells and about 2.8 billion ST_Intersects calls in a nested
loop. -link-pylovo spent 29 minutes in this query before linking
started.

Each building's cell is now floor(x / size), floor(y / size) of a point
inside its footprint, and batches are grouped by that index. The cells
are the same origin-aligned squares, every building still lands in
exactly one batch, and empty cells never exist. On the same database
the query returns the same 49 batches in 39 ms.
@codecov

codecov Bot commented Oct 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jravani
jravani merged commit 2b77b23 into main Oct 22, 2026
8 checks passed
@jravani
jravani deleted the perf/link-grid-batches branch October 22, 2026 16:08
@jravani jravani mentioned this pull request Oct 23, 2026
@jravani jravani added this to the v0.12.0 milestone Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant