New apache/iggy-demos repo: Apache producer/consumer Kafka client and Iggy demos

[justin-laserdata]

Hi,

We now have a separate repo for demos: https://github.com/apache/iggy-demos. Each demo lives in its own folder so it stays out of the main repo’s CI.

The first demo I put up is useful for two reasons, and it's more than just a demo. See https://github.com/apache/iggy-demos/tree/main/kafka-clients.

It implements Apache Kafka pub/sub on Iggy. The kafka-clients folder implements Kafka’s Producer, Consumer and Admin interfaces on top of the Iggy Java SDK, most of what you need is implemented. You change the constructor in you Kafka code and the rest of your Kafka client code stays the same. It works with kafka-clients 4.3 and 3.9 and stores records the same way the Kafka gateway does, so you can read records written by either one. I tested this up to the 1,000,000 messages per second range and also ported Camel, Debezium, NiFi, Kafka examples, Flink, Pinot, and Zipkin Kafka programs to see if it works and how much work it is - teh answer is it does work, and it's not a lot of effort. This won't give you the feature the Kafka Gateway provides, but it might be a path for people who want to integrate Kafka and Iggy.

To compare with the standard Iggy SDK, I also created some visual demos. One local web page has a tab each for the Iggy Java SDK, the Iggy Rust SDK, and the Kafka clients. Each tab runs producers and consumers and charts throughput and end-to-end latency live, so you can see what happens when producers outrun a consumer, what happens when you change message size and other settings live. I've added safeguards to prevent you from running out of disk space in Docker, but it may still be possible. A fourth tab starts a three-node Iggy cluster in Docker, where you can stop a node, or the leader, and watch the cluster carry on.

The Kafka part isn’t feature complete yet. It has no transactions, and consumer group offsets aren’t stored the way the gateway stores them. The README lists the limits.

Please try it, and if you have a demo or use case, open a pull request with a new folder.

Kind regards,
Justin