Add opt-in active discovery of cluster name to OpenTelemetry For Elasticsearch instrumentation.

Description

Cross-posted from open-telemetry/opentelemetry-java-instrumentation#19376.

Problem

When monitoring a self-managed Elasticsearch cluster with OpenTelemetry, the spans this client produces and the metrics the Collector's elasticsearchreceiver produces cannot be correlated, because they identify the cluster differently:

elasticsearchreceiver tags metrics with elasticsearch.cluster.name, from the cluster's monitoring API.
This client's spans (via OpenTelemetryForElasticsearch) never carry a cluster name at all.

I looked through OpenTelemetryForElasticsearch.java to confirm what's actually captured today. Per request, it sets:

db.system.name = "elasticsearch" (static)
db.operation.name (the endpoint id)
http.request.method, http.response.status_code
url.full, server.address, server.port (from httpResponse.node().uri())
db.elasticsearch.path_parts.* (per path parameter, via AttributeKeyTemplate)

None of these identify the cluster. server.address/server.port reflect whichever node (or load balancer / proxy) the low-level transport happened to hit — in most production topologies this is not the cluster's own identity, and Elasticsearch's data-plane responses (_search, _bulk, etc.) never return cluster_name for the client to pick up passively.

solution

An opt-in, off-by-default active discovery step in OpenTelemetryForElasticsearch, following the same pattern already used for otel.instrumentation.elasticsearch.capture-search-query:

New config option, e.g. otel.instrumentation.elasticsearch.capture-cluster-name (system property / OTEL_INSTRUMENTATION_ELASTICSEARCH_CAPTURE_CLUSTER_NAME env var), default false.
When enabled, on first use of a given transport/node pool, issue a single GET / and cache the cluster_name field from the response, scoped to that client instance.
Stamp an attribute carrying that value onto every subsequent span from that client — I've suggested db.elasticsearch.cluster.name below (following the existing db.elasticsearch.path_parts naming convention), but I'm flexible on the exact attribute name/namespace. I know the DB semantic conventions have been under active revision recently (this file itself already tracks that, e.g. DB_SYSTEM_NAME rather than the older DB_SYSTEM), so happy to align to whatever's current/settled rather than debate naming here.
On failure (403, timeout, older ES version quirks) fail silently and fall back to today's behavior — never block or fail the actual request.

Because the plumbing for Elasticsearch-specific attributes already exists (AttributeKeyTemplate, PATH_PART_PREFIX), this would be a small, idiomatic addition rather than a new mechanism.

alternatives considered

Passively picking up cluster name from a normal response. Doesn't work — data-plane endpoints (_search, _bulk) never return cluster_name.
Correlating on server.address in the Collector. Breaks under any load balancer, proxy, or container network — which is most production self-managed deployments. This is exactly what server.address reflects today, and it's not a reliable join key.
Manual tagging on both APM and Collector config. Requires two separate teams to coordinate and hand-maintain matching strings — defeats the point of automatic correlation.

Happy to help test/validate against a self-managed cluster if useful.

@vagrawal-newrelic

Hey @vagrawal-newrelic! Instead of an additional opt in metric, how about an extension mechanism that allows custom metrics to be added? This is what the .net client does already, I'm figuring it would work better long term. What do you think?

@l-trotta

Thanks @l-trotta , agreed on going the extension route rather than per-feature flags. A provider interface means new attributes can come from new providers instead of a new flag each time, which is the reusable direction I'd rather build.

The way I'm picturing it: the instrumentation exposes a small provider interface for contributing attributes to a span, and Elastic ships one default provider that fills in the cluster info. Anyone who needs a custom attribute writes their own provider and registers it the same way, without new flags or core changes. Any options (like a refresh interval) would just be settings on the provider when you register it, rather than global config.

Before I go further, a few design points I'd like us to agree on for that default cluster-info provider:

Where the cluster name comes from. The .NET client reads it from the X-Found-Handling-Cluster header, which is added by the Elastic Cloud proxy. Self-managed clusters don't send it today, and normal responses (_search, _bulk, …) don't carry cluster_name. So I'd have the provider check the header first and only fall back to an active GET / when it's absent. That keeps it future-proof: if the header ever becomes available for self-managed too, the provider picks it up automatically with no change on our side, and the active fetch just stops being needed.
How the active fetch is made. That GET / should go through the client's own transport rather than a separate HTTP client, so it inherits the configured auth, TLS/mTLS, proxy and node selection and doesn't break on secured clusters. Two things I'd want your take on: is it OK for a provider to access the transport for this, and is there a preferred way to keep that internal request from being instrumented / looping back on itself?
Caching and refresh. The provider would fetch once and cache per client, then refresh on an interval (I'd suggest a 30-minute default, configurable) since the name can change when a node restarts. Any failure (403, timeout, older versions) would fall back silently and never affect the real request. Do those defaults sound reasonable?
What it captures. db.elasticsearch.cluster.name matches what .NET already defines, so that's the natural primary attribute. Since the GET / returns cluster_uuid in the same response, the provider could expose that too at no extra cost, giving a stable unique identifier for anyone who wants one later. I'd suggest the default captures the name, with uuid as an easy opt-in.

If this lines up with what you had in mind, I'll draft it and open a PR for review.

@vagrawal-newrelic

@vagrawal-newrelic I took a better look at the current situation for the java client, and realized that an extension mechanism is already in place, and you can add cluster name (or any custom metric) by adding a custom SpanProcessor:

SpanProcessor esSpanEnricher = new SpanProcessor() {
    @Override public void onStart(Context ctx, ReadWriteSpan span) {
        if ("elasticsearch-api".equals(span.getInstrumentationScopeInfo().getName())) {
            span.setAttribute(
                AttributeKey.stringKey("db.elasticsearch.cluster.name"),
                clusterName
            );
        }
    }
    @Override public boolean isStartRequired() { return true; }
    @Override public void onEnd(ReadableSpan span) {}
    @Override public boolean isEndRequired() { return false; }
};

OpenTelemetry customOtel = OpenTelemetrySdk.builder()
    .setTracerProvider(SdkTracerProvider.builder()
        .addSpanProcessor(esSpanEnricher)
        // .addSpanProcessor(your exporter processor here)
        .build())
    .build();


OpenTelemetryForElasticsearch esOtelInstrumentation =
    new OpenTelemetryForElasticsearch(customOtel, false);

try (ElasticsearchClient esClient = ElasticsearchClient.of(b -> b
    .host(serverUrl)
    .apiKey(apiKey)
    .instrumentation(esOtelInstrumentation)
)) {

you can use this workaround while I decide if this is actually worth implementing and how :) I'll keep this open in the meantime!

@l-trotta

Thanks @l-trotta , that's a helpful pointer — the custom SpanProcessor is a clean way to attach the attribute without touching the client.

The limitation is that it needs the cluster name supplied up front (hardcoded), with each user wiring up their own processor and discovery. For anything meant to work generically / out of the box, that per-setup custom code isn't really viable — which is where having the client resolve the name itself would make the difference: it would work for any user without hand-rolling anything.

The core gap is dynamically discovering the name, particularly for self-managed, where it isn't available passively: data-plane responses (_search, _bulk) don't return cluster_name, and the X-Found-Handling-Cluster header is Elastic Cloud only.

A native default could:

Check the X-Found-Handling-Cluster header first (so if it's ever added for self-managed too, it keeps working with no change).
Fall back to a single GET / through the client's own transport (reusing the configured auth/TLS) to resolve the name.
Cache the result with a refresh interval.
Attach db.elasticsearch.cluster.name automatically (and cluster_uuid from the same response as a bonus).

Happy to open a PR along these lines if you think it's worth adding. No rush, and thanks for keeping it open while you evaluate!

Capture the Elasticsearch cluster name on spans via a pluggable provider

@l-trotta

Hi @l-trotta — to make the goal and approach concrete, I've opened a draft PR (#1319) as a starting point for discussion. I'd really value your feedback on the direction before going further.

Happy to adjust the design, attribute names, or approach based on your guidance — thanks for taking a look!

@vagrawal-newrelic

hello @vagrawal-newrelic, thanks for your proposal! we've been discussing with both my team and the APM team, we're trying to figure out a more generic approach that would work for all clients. when we finalize the details we can reiterate on your PR if you'd like to keep working on this.

@vagrawal-newrelic

Hey @vagrawal-newrelic! I have good news: we decided to take a more general approach, so it will be the server itself that will provide the on-prem cluster name as a header, just like it already does with Cloud. The server side PR has already been merged, so now it's just a matter for the java client to handle both headers. Would you be willing to modify your PR to fit the new requirements?

Here's a short summary of the work that should be done:
Update the client's OpenTelemetry instrumentation to read the X-Found-Handling-Cluster and the new Elastic-Cluster-Name response headers and record the cluster name (db.elasticsearch.cluster.name). Precedence when both are present: prefer X-Found-Handling-Cluster, since its presence indicates an Elastic Cloud (proxied) deployment and it carries the canonical, globally-unique cluster resource id. Fall back to Elastic-Cluster-Name when only it is present (the self-managed / on-prem case).

@l-trotta

Thanks a lot @l-trotta for the quick turnaround and for laying out the approach and naming so clearly — really appreciate it! 🙏
That all makes sense to me. I'll go ahead and make the changes: read both headers, prefer X-Found-Handling-Cluster and fall back to Elastic-Cluster-Name, record it as db.elasticsearch.cluster.name, and stay quiet when neither is present. I'll get the PR up and validate it against a self-managed cluster.

@l-trotta

@l-trotta the client-side change is up and ready for review: #1319

It follows the approach you outlined — it reads the cluster name from the response headers and stamps db.elasticsearch.cluster.name, preferring X-Found-Handling-Cluster (Elastic Cloud) and falling back to Elastic-Cluster-Name (self-managed, 9.6+). The value is read per response, so a cluster rename is reflected without restarting the client, and it makes no extra request. It includes unit tests covering all the header combinations and a short note in the OpenTelemetry docs; ./gradlew check passes locally.

Since I'm an external contributor, the CI workflows are waiting on maintainer approval to run — would you mind kicking those off and taking a look when you get a chance? Happy to adjust anything. Thanks!

Navigation Menu

Repository navigation

Add opt-in active discovery of cluster name to OpenTelemetry For Elasticsearch instrumentation. #1304

Description

Problem

solution

alternatives considered

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add opt-in active discovery of cluster name to OpenTelemetry For Elasticsearch instrumentation. #1304

Description

Description

Problem

solution

alternatives considered

Activity

l-trotta commented on Aug 5, 2026

vagrawal-newrelic commented on Aug 6, 2026

l-trotta commented on Aug 6, 2026

vagrawal-newrelic commented on Aug 7, 2026

vagrawal-newrelic commented on Aug 17, 2026

l-trotta commented on Aug 18, 2026

l-trotta commented on Sep 3, 2026

vagrawal-newrelic commented on Sep 4, 2026

vagrawal-newrelic commented on Sep 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions