Possible connection handling regression in Elastic.Clients.Elasticsearch 8.17 causing CPU spikes

[MychellSantos]

Elastic.Clients.Elasticsearch version: 8.17.4 (previously 8.15.0)

Elasticsearch version: Elastic Cloud (SaaS) 8.15 (before upgrade) — issue noticed after upgrading the client to 8.17

.NET runtime version: .NET 9.0

Operating system version: Linux (Kubernetes pods) — 10 pods handling connections

Description of the problem including expected versus actual behavior:
I am investigating a possible performance degradation after upgrading the Elastic.Clients.Elasticsearch library from version 8.15 to 8.17.

During load testing, I noticed an unusual increase in CPU usage in the Elasticsearch cluster when running queries. I suspect this might be related to open connections or changes in how the Elastic.Transport layer handles connections between these versions.

This was not observed with the 8.15 client version.

Steps to reproduce:

Deploy an application using .NET 9.0 with Elastic.Clients.Elasticsearch 8.17.0.

Set up 10 pods handling a load of approximately 1.3k requests per minute.

Perform multiple search queries under load and monitor CPU usage on the Elastic Cloud cluster.

Expected behavior:
Connection handling and resource usage similar to previous versions (8.15), with stable CPU usage on the Elasticsearch cluster under the same load conditions.

Provide ConnectionSettings (if relevant):
Using the default settings provided by the Elastic.Clients.Elasticsearch client for 8.17, configured to connect to Elastic Cloud SaaS. (If needed, I can share more detailed configuration.)

Provide DebugInformation (if relevant):
I can provide logs and debug output if necessary to help diagnose the issue.

[flobernd]

Hi @MychellSantos , please upgrade to client version 8.17.4 or later. This issue sounds like a duplicate of #8462.

I would recommend to always use the latest patch version of a given {major}.{minor} version (8.17.4 at the moment) or even the latest {major}.{minor} (8.18.1 at the moment) as these come with the highest chance to already include a bugfix for a given problem.

The minor versions of the client are (99.9% of the times) backward compatible to all earlier Elasticsearch minor versions of the same major version (see here: compatibility).

Please let me know if that fixes your issue.

[MychellSantos]

Hi @flobernd ,

Thank you very much for being so quick and precise. In a sea of possibilities, it's great to know we can navigate a less nebulous path with your help.

I'm planning to upgrade to version 9 soon. At the time I upgraded the client, version 8.17.4 was the latest available.

I'm still considering the possibility that the issue might be related to the semantic search feature we're working on. We also upgraded to this version to implement the K parameter in k-NN.

I will run tests both with and without semantic search enabled to verify this, and I'll do my best to keep you updated on the results.

Thanks again!

[flobernd]

Hi @MychellSantos , no problem, happy to help 🙂

Besides that, I think I misread the problem. I assumed you were talking about CPU usage on client side but you clearly mentioned CPU usage on the server/cluster.

Given that and the fact that you already used 8.17.4 means that what you observe is not related to #8462.

The client itself does not change anything in terms of connection handling, serialization, etc. that could affect an Elasticsearch server/cluster in the described way. I strongly assume that indeed the different workload causes a higher CPU utilization, which might be a completely normal behavior (some features require more resources than others).

To completely rule out the client as the source of this issue, it would be good, if you could test the exact same workload with the 8.15.x and the 8.17.4 client.

It might as well be worth to contact our support to learn more about the CPU usage / resource requirements of specific Elasticsearch features (I sadly can't help you with that) or ask about this in the discuss forum.