Enrich cache default can usage a ton of memory

[nik9000]

Elasticsearch Version

8.16+

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

In 8.16 we changed the default for enrich.cache_size from 1000 to 1% which should allow us to more carefully cache. The trouble is that we don't account for a few things in the cache's weight:

1. The cache key. I'm seeing those in the 600 bytes size.
2. Empty results.
3. Results use their serialized size but are then stored as Map<String, Object> which is usually much larger.
4. It sure looks like there isn't any intrinsic cost to a cache entry. Like, for the entry itself.

When you combine the first two, queries pointing to no documents in the enrich index cost 0 bytes. So, no matter the limit you configure, you can fill up memory with them.

But, yeah, I think we should include the size of the cache key. And Include an estimate of the cost of the List we store in the cache - even if it's empty. And, finally, I think we should uplift the cost of the exploded map. Or we should cache the serialized copy. I'm not entirely sure if it's worth a cost for the cache entry itself. That bit I don't know.

Steps to Reproduce

See above.

Logs (if relevant)

@nik9000 has a heap dump of a node the OOMed full of cache entries of empty results lists.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)