From 8a016af46dff6e94944231c968373955bc46a0ba Mon Sep 17 00:00:00 2001 From: pierwill <19642016+pierwill@users.noreply.github.com> Date: Fri, 22 Apr 2022 10:21:06 -0500 Subject: [PATCH] Add instructions on increasing `max_map_count` (#3932) * Add instructions on increasing max_map_count * Update frequently-asked-questions.md Co-authored-by: pierwill Co-authored-by: Scott Anderson --- .../frequently-asked-questions.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/content/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions.md b/content/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions.md index cc651f0ce4..6f1807971f 100644 --- a/content/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions.md +++ b/content/enterprise_influxdb/v1.9/troubleshooting/frequently-asked-questions.md @@ -89,6 +89,7 @@ Where applicable, it links to outstanding issues on GitHub. * [Why am I seeing `error writing count stats ...: partial write` errors in my data node logs?](#why-am-i-seeing-error-writing-count-stats--partial-write-errors-in-my-data-node-logs) * [Why am I seeing `queue is full` errors in my data node logs?](#why-am-i-seeing-queue-is-full-errors-in-my-data-node-logs) * [Why am I seeing `unable to determine if "hostname" is a meta node` when I try to add a meta node with `influxd-ctl join`?](#why-am-i-seeing-unable-to-determine-if-hostname-is-a-meta-node-when-i-try-to-add-a-meta-node-with-influxd-ctl-join) +* [Why is InfluxDB reporting an out of memory (OOM) exception when my system has free memory?](#why-is-influxdb-reporting-an-out-of-memory-oom-exception-when-my-system-has-free-memory) --- @@ -1333,3 +1334,35 @@ Meta nodes use the `/status` endpoint to determine the current state of another `"nodeType":"meta","leader":"","httpAddr":":8091","raftAddr":":8089","peers":null}` If you are getting an error message while attempting to `influxd-ctl join` a new meta node, it means that the JSON string returned from the `/status` endpoint is incorrect. This generally indicates that the meta node configuration file is incomplete or incorrect. Inspect the HTTP response with `curl -v "http://:8091/status"` and make sure that the `hostname`, the `bind-address`, and the `http-bind-address` are correctly populated. Also check the `license-key` or `license-path` in the configuration file of the meta nodes. Finally, make sure that you specify the `http-bind-address` port in the join command, e.g. `influxd-ctl join hostname:8091`. + +## Why is InfluxDB reporting an out of memory (OOM) exception when my system has free memory? + +`mmap` is a Unix system call that maps files into memory. +As the number of shards in an InfluxDB Enterprise cluster increases, the number of memory maps increase. +If the number of maps exceeds the configured maximum limit, the node reports that it is out of memory. + +To check the current number of maps the `influxd` process is using: + +```sh +# Get the influxd process ID (PID) +PID=$(ps aux | awk '/influxd/ ${print 2}' + +# Count the number of maps associated with the influxd process +wc -l /proc/$PID/maps +``` + +The `max_map_count` file contains the maximum number of memory map areas a process may have. +The default limit is `65536`. +We recommend increasing this to `262144` (four times the default) by running the following: + +```sh +echo vm.max_map_count=262144 > /etc/sysctl.d/90-vm.max_map_count.conf +``` + +To make the changes permanent: + +```sh +sysctl --system +``` + +Restart the `influxd` process and repeat on each node in your cluster.