You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(connectors): OTLP/gRPC source connector for logs, metrics and traces #3815
Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.
OTLP is listed as a P0 source and sink on the connector roadmap (#2753).
Affected area / component
Connectors
Proposed solution
An otlp_source connector plugin that binds the OTLP standard port and serves LogsService, MetricsService and TraceService from a single listener, since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.
Decode with opentelemetry-proto, which already ships the generated tonic
stubs, so no build.rs or protoc enters the build.
Two storage formats:
json, one queryable document per record, for sinks that need field-level
access.
proto, the raw request bytes as a single message, roughly 4-5x smaller and
zero-copy when paired with an OTLP sink.
Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's partial_success, counted
in OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.
Alternatives considered
Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.
Generating the protobuf stubs with tonic-build. It adds protoc to the build
for types opentelemetry-proto already publishes.
Hi, @mfyuce . I've already finished sink connector. I build that partly based on your code. I'll co-author you on sink connector. If you not comfortable with it, tell me about it.
You can take source connector if you have time to work on it.
Activity
ryankert01 commentedon Sep 7, 2026
hi @mfyuce , are you still interested on continuing the work? I can help take over if you're no longer interested.
justinmclean commentedon Oct 22, 2026
@ryankert01 nobody is assigned to this, so it is yours if you still want it.
mfyuce commentedon Oct 28, 2026
Hello @ryankert01 and @justinmclean
I really want to continue and infact doing it on a very big data locally.
Do i have time to go over it? I need a month or more for my tests to really pay off.
Continuing on the prev PRs and taking lessons from them.
ryankert01 commentedon Oct 28, 2026
Hi, @mfyuce . I've already finished sink connector. I build that partly based on your code. I'll co-author you on sink connector. If you not comfortable with it, tell me about it.
You can take source connector if you have time to work on it.
mfyuce commentedon Oct 29, 2026
hello @ryankert01,
Really love to.
Thanks!