Skip to content

feat(connectors): OTLP/gRPC source connector for logs, metrics and traces #3815

Open

Description

@mfyuce

Description

Getting OpenTelemetry data into Iggy currently means running a separate bridge
process in front of it. Collectors and SDKs already speak OTLP/gRPC, and the
connectors runtime already hosts long-lived plugins, so the bridge is avoidable:
the OTLP service can terminate inside the runtime and produce straight onto Iggy
topics.

OTLP is listed as a P0 source and sink on the connector roadmap (#2753).

Affected area / component

Connectors

Proposed solution

An otlp_source connector plugin that binds the OTLP standard port and serves
LogsService, MetricsService and TraceService from a single listener, since
standard OTLP deployments expose one endpoint and splitting them would force
operators to configure three addresses that match no existing tooling.

Decode with opentelemetry-proto, which already ships the generated tonic
stubs, so no build.rs or protoc enters the build.

Two storage formats:

  • json, one queryable document per record, for sinks that need field-level
    access.
  • proto, the raw request bytes as a single message, roughly 4-5x smaller and
    zero-copy when paired with an OTLP sink.

Backpressure must not fail the export. When the in-process channel fills, enqueue
what fits and report the remainder in the response's partial_success, counted
in OTLP records rather than internal messages, so a client retries only what was
actually rejected instead of resending a batch that was half delivered.

Alternatives considered

Running the OpenTelemetry Collector with a custom exporter. That keeps a second
process in the path and duplicates the connector runtime's lifecycle, config and
metrics handling.

Generating the protobuf stubs with tonic-build. It adds protoc to the build
for types opentelemetry-proto already publishes.

Contribution

Yes, I am willing to submit a PR.

Activity

  1. ryankert01 commented on Sep 7, 2026

    @ryankert01
    Member

    hi @mfyuce , are you still interested on continuing the work? I can help take over if you're no longer interested.

  2. justinmclean commented on Oct 22, 2026

    @justinmclean
    Member

    @ryankert01 nobody is assigned to this, so it is yours if you still want it.

  3. mfyuce commented on Oct 28, 2026

    @mfyuce
    ContributorAuthor

    Hello @ryankert01 and @justinmclean

    I really want to continue and infact doing it on a very big data locally.

    Do i have time to go over it? I need a month or more for my tests to really pay off.

    Continuing on the prev PRs and taking lessons from them.

  4. ryankert01 commented on Oct 28, 2026

    @ryankert01
    Member

    Hi, @mfyuce . I've already finished sink connector. I build that partly based on your code. I'll co-author you on sink connector. If you not comfortable with it, tell me about it.

    You can take source connector if you have time to work on it.

  5. mfyuce commented on Oct 29, 2026

    @mfyuce
    ContributorAuthor

    hello @ryankert01,

    Really love to.

    Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions