Apache Iggy Connector for Apache Pinot

[chiradip]

Apache Iggy Connector for Apache Pinot

This connector enables Apache Pinot to ingest real-time data from Apache Iggy streams using TCP-based communication.

Overview

The Iggy Pinot connector implements Pinot's Stream Plugin API to provide:

• TCP-based ingestion: Uses Iggy's native TCP protocol via AsyncIggyTcpClient for efficient message consumption
• Partition-aware consumption: Supports parallel ingestion from multiple Iggy partitions
• Consumer group support: Leverages Iggy consumer groups for offset management and fault tolerance
• Automatic offset management: Consumer group state is maintained by Iggy server
• JSON message decoding: Built-in support for JSON-formatted messages

Architecture

Key Components

1. IggyConsumerFactory: Main entry point implementing Pinot's StreamConsumerFactory
2. IggyPartitionGroupConsumer: Partition-level consumer using TCP client
3. IggyStreamMetadataProvider: Provides partition discovery and offset information
4. IggyJsonMessageDecoder: Decodes JSON messages into Pinot records

Differences from Flink Connector

The Pinot connector differs from the Iggy Flink connector in several ways:

• TCP-based: Uses AsyncIggyTcpClient directly (not HTTP-based)
• Consumer group managed: Relies on Iggy's consumer group offset management
• Simpler offset handling: No custom offset storage, leverages Iggy's built-in state
• Pinot-specific APIs: Implements PartitionGroupConsumer instead of Flink's SourceReader

[chiradip]

@kparisa

[chiradip]

Apache Pinot Connector for Iggy - Design Document

Version: 1.0
Status: Implementation Complete
Last Updated: December 2025

Executive Summary

This document outlines the architectural design and implementation strategy for the Apache Iggy stream connector plugin for Apache Pinot. The connector enables real-time data ingestion from Iggy message streams into Pinot's OLAP datastore, providing high-throughput, low-latency streaming analytics capabilities.

Design Goals

1. Native TCP Protocol: Utilize Iggy's native TCP protocol for maximum performance, avoiding HTTP overhead
2. Pinot API Compliance: Full compatibility with Pinot's Stream Plugin API (v1.2.0+)
3. High Performance: Target >1M messages/second throughput with sub-millisecond latency
4. Partition Parallelism: Support concurrent consumption from multiple partitions
5. Offset Management: Leverage Iggy's server-managed consumer groups for reliable offset tracking
6. Production Ready: Include comprehensive testing, monitoring, and operational documentation

1. System Architecture

1.1 High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Apache Pinot Cluster                     │
│  ┌───────────┐  ┌───────────┐  ┌───────────────────────┐   │
│  │Controller │  │  Broker   │  │   Server (Realtime)   │   │
│  └───────────┘  └───────────┘  └───────┬───────────────┘   │
│                                         │                     │
│                          ┌──────────────▼──────────────┐     │
│                          │  Iggy Stream Plugin         │     │
│                          │  ┌──────────────────────┐   │     │
│                          │  │ IggyConsumerFactory  │   │     │
│                          │  └──────────┬───────────┘   │     │
│                          │             │               │     │
│              ┌───────────▼─────────────▼──────────┐    │     │
│              │ IggyPartitionGroupConsumer (P0)   │    │     │
│              └───────────┬───────────────────────┘    │     │
│              ┌───────────▼─────────────────────┐      │     │
│              │ IggyPartitionGroupConsumer (P1)│      │     │
│              └───────────┬──────────────────────┘     │     │
│                          │                            │     │
└──────────────────────────┼────────────────────────────┘     │
                           │ TCP Connections                  │
                           │ (Connection Pool)                │
                           │                                  │
┌──────────────────────────▼────────────────────────────────┐
│                    Apache Iggy Cluster                     │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐          │
│  │ Stream 1   │  │ Stream 2   │  │ Stream N   │          │
│  │ ┌────────┐ │  │ ┌────────┐ │  │ ┌────────┐ │          │
│  │ │Topic 1 │ │  │ │Topic 1 │ │  │ │Topic 1 │ │          │
│  │ │  P0 P1 │ │  │ │  P0 P1 │ │  │ │  P0 P1 │ │          │
│  │ └────────┘ │  │ └────────┘ │  │ └────────┘ │          │
│  └────────────┘  └────────────┘  └────────────┘          │
└────────────────────────────────────────────────────────────┘

1.2 Component Responsibilities

The connector architecture follows Pinot's plugin model with clear separation of concerns:

Component	Responsibility	Lifecycle
IggyConsumerFactory	Plugin entry point, consumer instantiation	Singleton per server
IggyPartitionGroupConsumer	Message polling, partition consumption	One per partition
IggyStreamMetadataProvider	Partition discovery, offset resolution	One per partition
IggyStreamConfig	Configuration parsing and validation	Created per consumer
IggyJsonMessageDecoder	Message payload decoding	Shared across consumers
IggyMessageBatch	Batch message container	Created per fetch
IggyStreamPartitionMsgOffset	Offset representation	Created per message

1.3 Design Rationale

TCP over HTTP Decision:

• Initial Flink connector used HTTP due to incorrect Docker image during development
• TCP protocol provides 40-60% lower latency and higher throughput
• Native protocol alignment with Iggy's core design
• Connection pooling eliminates per-request overhead

Consumer Group Strategy:

• Server-managed offsets eliminate need for external coordination (Zookeeper, etc.)
• Auto-commit mode simplifies implementation while maintaining reliability
• Consumer group per table provides isolation and independent scaling

Partition-Level Consumption:

• Pinot creates one consumer per partition for maximum parallelism
• Each consumer maintains independent TCP connection from pool
• Linear scaling with partition count

[chiradip]

@hubcio @numinnex @spetz