Skip to content

Cut allocation in GeoPackage reads: one row at a time, zero-copy geometry - #237

Merged
jampukka merged 1 commit into
nlsfi:mainfrom
jampukka:perf/source-gpkg-read-path
Oct 16, 2026
Merged

jampukka merged 1 commit into
nlsfi:mainfrom
jampukka:perf/source-gpkg-read-path

Conversation

@jampukka

Copy link
Copy Markdown
Collaborator

This is purely a performance optimization of the hakunapi-source-gpkg read path: it lowers allocation and GC pressure per feature. Output is unchanged and no new configuration is added.

  • Row-at-a-time reading (concept, not a class)
    • One row per hasNext() into a single reused container, instead of batches of 250. SQLite runs in-process, so a batch saves no round trips, and it keeps every row's values (geometry byte[] above all) reachable until the batch is consumed.
    • Precondition for the zero-copy geometry below: that buffer is only valid until the cursor advances.
  • Zero-copy geometry access
    • A geometry blob is read through a direct ByteBuffer over SQLite's own memory instead of a new byte[] per feature. The buffer, the HakunaGeometryGPKG built on it and the ValueProvider are all valid only until the next hasNext().
  • ResultSetFeatureStream
    • Streams the query's rows as above. Owns the statement, result set and connection, and closes them.
  • ResultSetValueProvider
    • Still the JDBC-backed provider. isNull asks the column's storage class and builds no value, and getHakunaGeometry uses SqliteBlobReader when one is available.
  • SqliteBlobReader (reflection)
    • Obtained once per query from the PreparedStatement. Null when the sqlite-jdbc internals it needs are not found or the statement is not sqlite-jdbc's, in which case getBytes is used.
  • SqliteColumnType
    • Reads the current row's storage class from the driver's column_type, which is zero-based unlike JDBC's one-based indexes. Unwraps the statement because the connection pool hands out a proxy. Null when the statement is not SQLite's, in which case isNull uses getObject.
  • hakunapi-source-postgiskeeps its own copy ofBufferedResultSet`.
  • SqliteBlobReader falls back to getBytes only when its reflective lookups fail, not when a sqlite-jdbc upgrade changes what those internals do. SqliteColumnType compiles against org.sqlite.core, so an incompatible upgrade fails at build time, but a change in what the calls return is not detected.

@jampukka
jampukka force-pushed the perf/source-gpkg-read-path branch from 526721b to 91deb0a Compare October 16, 2026 16:53
@jampukka
jampukka merged commit 1e46563 into nlsfi:main Oct 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant