Skip to content

Support Apache Flink 2.x (currently capped at Flink 1.19) #3053

Description

@joshmanis618-stack

Problem

Sedona's Flink module is locked to Flink 1.x — 1.9.0 targets Flink 1.19, and the
latest dependency work (#2985) only moves to flink-table-planner_2.12 1.20.4,
still Flink 1.x and still Scala-suffixed. There's no path to run Sedona on Flink
2.x today.

This blocks teams standardizing on Flink 2.x — e.g. pipelines that need 2.x-era
Kafka source / deserialization APIs (Protobuf ingestion) can't drop below 2.0,
so Sedona's spatial functions are simply off the table. My use case is streaming
map-matching / spatial joins of vehicle positions against an indexed road network
— a spatial-index workload that doesn't reduce to point-level shapely calls.

Scope

The geometry core (sedona-common) is Flink-agnostic; the break is confined to
sedona-flink:

  • Scala-free artifactssedona-flink_2.12 / flink-table-planner_2.12
    the unsuffixed Flink 2.x table artifacts.
  • Removed legacy interfacesSourceFunction / SinkFunction / SinkV1 are
    gone in 2.0; move to Source/Sink v2.
  • Serialization — Geometry TypeInformation / serializer registration likely
    needs updating for Flink 2.x.
  • Build — a Flink 2.x variant alongside the 1.x line.

Questions

  1. Is Flink 2.x support already on anyone's radar / in progress?
  2. Would a Flink-2.x compatibility module be accepted as a contribution (kept
    side-by-side with 1.x during transition), or is there a preferred shape first?

Happy to test against a real Flink 2.2 streaming workload if someone picks it up.

Environment

  • Sedona 1.9.0 · Flink target 2.2.0 · streaming Table/DataStream, geospatial map-matching

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions