make kafka topic creation and partition config configurable via env vars - #255
make kafka topic creation and partition config configurable via env vars#255VINODvoid wants to merge 11 commits into
Conversation
|
@haileyok hey, took a stab at addressing the review feedback from @ThisIsMissEm and @chimosky — opened a branch off yours if you want to take a look! |
|
wow, thank you! I'll take a look at this later today, very much appreciate this!!! |
| output_topic = config.expect_str('OSPREY_KAFKA_OUTPUT_TOPIC') | ||
| bootstrap_servers = config.expect_str_list('OSPREY_KAFKA_BOOTSTRAP_SERVERS') | ||
| client_id = config.expect_str('OSPREY_KAFKA_OUTPUT_CLIENT_ID') | ||
| auto_create_topic = config.get_bool('OSPREY_KAFKA_AUTO_CREATE_TOPIC', True) |
There was a problem hiding this comment.
In your opening comment, you've said "OSPREY_KAFKA_AUTO_CREATE_TOPIC (default: true) — set to false for deployments that manage Kafka topics via Terraform or other IaC tooling.", but you've set the default here to True and also set auto_create_topic to True.
|
Hey @VINODvoid, thanks for this contribution! We took another look in our recent working group call, and the consensus is that we do want to see this land. Are you able to resolve conflicts and address the feedback so far? |
|
@VINODvoid Are you still interested in finishing this PR, or should someone else from the community take it on? Thanks! |
|
@reitblatt Yes, still interested; happy to finish it. But I want to check direction first, because I think this is blocked on more than the conflicts. This is based on |
|
I'd made some changes here, you can take a look, tests fail but I just pushed and haven't fixed the tests. |
|
@chimosky Can u point to the test , link is not pointed there |
5 participants
Description
Addresses two open review comments on #220.
KafkaOutputSinkhad hardcodednum_partitions=3andreplication_factor=3inensure_topic(), and topic auto-creation always ran unconditionally on startup. This makes bothconfigurable via environment variables.
OSPREY_KAFKA_AUTO_CREATE_TOPIC(default:true) — set tofalsefor deployments that manage Kafka topics via Terraform or other IaC tooling.OSPREY_KAFKA_NUM_PARTITIONS(default:1) andOSPREY_KAFKA_REPLICATION_FACTOR(default:1) — replaces the hardcoded3values. Defaults to1so single-broker localsetups (docker-compose) don't fail silently on topic creation. Production deployments can override via env.
Both are read in
sink_register.pyand passed intoKafkaOutputSink.__init__, following the same pattern as all other Kafka config in this codebase.Branch based on
hailey/confluent-kafka.Checklist
uv run ruff check .passes (no unused imports or other lint errors)uv tool run fawltydeps --check-unused --pyenv .venvpasses (no unused dependencies)CHANGELOG.mdwith my changes, if applicable