This repository demonstrates a basic event pipeline from a client sending POST requests to an API which forwards messages into Kafka topics for workers to process and store in PostgreSQL, with a Grafana telemetry stack.
The project may be run locally in 2 ways
- Docker Compose
- Kubernetes Cluster
The compose approach is much simpler in that it requires fewer tools to run and it consists of 1 instance of each component (load generator, api, kafka, worker, postgresql).
The Kubernetes setup needs extra tooling (kind, kubectl, Helm, KEDA, and related scripts). You can scale the load generator with just scale-client <N> to simulate more clients, build Kafka consumer lag, and let KEDA scale workers automatically from 1 up to 4.
If you're interested in running the project take a look at the usage file.
More information about the repository and scope of the project may be found here.