Repository navigation
ES Java Client 9.2.4 -> 9.4.4 performance regression. HttpComponents 5.4.4 -> 5.6 to blame? #1289
Description
Java API client version
9.4.4
Java version
25
Elasticsearch Version
9.4.4
Problem description
We just upgraded all our ES clusters from 9.2.4 -> 9.4.4. Server upgrade was buttery smooth, as always! :flexed_biceps:
However, upgrading the ES Java client from 9.2.4 to 9.4.4 we noticed a performance regression - up to 50ms added to an msearch request.
This is our typeahead service - here you can see our p95 jumping by around 30ms, our p95 jumped by around 50ms:
We saw same issue in our main search ES cluster with upgraded client too...
We though at first this might be related to the change to GZIP handling with the upgrade to Apache HTTP Client 5.6, but after putting on TRACE logging, that doesn't appear to be the case - gzip headers / compression was working in both directions as expected in both 9.2.4 and 9.4.4
We were able to remove this performance regression by keeping ES Client at 9.4.4 but pinning the transitive dependency on HTTPClient 5 in Gradle with force("org.apache.httpcomponents.client5:httpclient5:5.4.4")
For comparison, here's the 9.4.4 client deployed with Apache HTTP Client 5 pinned to 5.4.4 - no real difference in performance 🙃 :
Here's a couple of OTEL traces showing the performance regression is defo in the msearch client call:
9.2.4 client - ~3ms in the span over msearch:
9.4.4 client - ~62ms in the span over msearch:
We think that there may be some further changes required in the ES Java client to handle the dependency upgrade on HTTP Client 5.4.4 -> 5.6.1 other than just the change to GZIP handling?
Activity
peedeeboy commentedon Jul 29, 2026
For fun and profit, we also tested with
9.3.6client version - same issue. So we defo suspect the upgrade to HTTP Client 5.6 as that is in:>= 9.2.5>= 9.3.0>= 9.4.0as we understand it...
l-trotta commentedon Jul 29, 2026
Hello, thanks a lot for the detailed report! I'm going to investigate and I'll get back as soon as I find out something, unfortunately the apache client update isn't something we can revert, because it would be causing breaking changes in libraries such as spring-data.
NikitaBonde20 commentedon Jul 31, 2026
Hello, please check for ES 9.1 as well, Thank you.
l-trotta commentedon Jul 31, 2026
hey @NikitaBonde20 for the vulnerabilities, parsson has been updated here #1273, so 9.1 didn't receive any update related to these 2 libraries.
l-trotta commentedon Jul 31, 2026
Back with good news :) it was an easy one, the regression was a bug in the httpcore (not sure when it was introduced) fixed in version 5.4.1. Version 5.6.1 of httpclient had a transitive dependency on httpcore 5.4.0, so updating to httpclient 5.6.3 (which uses httpcore 5.4.3) should be enough to fix this issue.
Many thanks again @peedeeboy for reporting this, who knows when I'd notice otherwise! I really should add a network and memory performance benchmark to the test suite.
I'll backport the fix up to 9.2 and will release emergency patches ASAP.
peedeeboy commentedon Jul 31, 2026
Thanks @l-trotta for your speedy resolution! Amazing work 💪