🏫 Awesome Stream Processing 🏫

The term "stream processing" might sound intimidating to many people. We often hear statements like:

• "Stream processing is too difficult to learn and use!" 😱
• "Stream processing is very expensive!" 😱
• "I don’t see any business use cases for stream processing!" 😱

However, we believe this isn't true. ❌

Streaming data is everywhere, generated from operational databases, messaging queues, IoT devices, and many other sources. People can leverage modern stream processing technology to easily address classic real-world problems, using SQL as the programming language.

In this repository, we provide a series of executable demos demonstrating how stream processing can be applied in practical scenarios:

0. Getting started ✅

   • Install Kafka, PostgreSQL, and RisingWave, and run minimal toy examples on your device.
   • Integrate RisingWave with other data platforms.

1. Basic stream processing examples ✅

   Learn the fundamentals of ingesting, processing, transforming, and offloading data from streaming systems.

   1. Querying and processing event streaming data (👈 Kafka users, you may start here! 💡)
   • Directly query data stored in event streaming systems (e.g., Kafka, Redpanda).
   • Continuously ingest and analyze data from event streaming systems.
   2. Bringing analytics closer to operational databases (👈 Postgres users, you may start here! 💡)
   • Offload event-driven queries (e.g., materialized views and triggers) from operational databases (e.g., MySQL, PostgreSQL).
   3. Real-time ETL (Extract, Transform, Load)
   • Perform ETL continuously and incrementally.

2. Simple demonstrations ✅

   • A collection of simple, self-contained demos showcasing how stream processing can be applied in specific industry use cases.

3. Solution demonstrations ✅

   • A collection of comprehensive demos showcasing how to build a stream processing pipeline for real-world applications.

We use RisingWave as the default stream processing system to run these demos. We also assume that you have Kafka and/or PostgreSQL installed and possess basic knowledge of how to use these systems. These demos have been verified on Ubuntu and Mac.

All you need is a laptop 💻 - no cluster is required.

Any comments are welcome. Happy streaming!

Join our Slack community to engage in discussions with thousands of stream processing enthusiasts!