Cannot connect to cluster running in docker

[mdorda]

Description

I am not able to connect to redis cluster running in docker. As far as I was able to debug it, it connects, it downloads cluster metadata and internal addresses and then it indefinitely hangs. Here is a minimal reproducible example:

Dockerfile for redis pod:

FROM redis:8.2.0

COPY entrypoint.sh /
RUN chmod +x /entrypoint.sh

CMD ["/entrypoint.sh"]

Its entrypount.sh

#!/bin/bash

REDIS_CONFIG_PATH=${REDIS_CONFIG_PATH:-/etc/redis.conf}

env | grep REDIS_CONF | grep -v REDIS_CONFIG_PATH | while read -r env_var; do
  redis_var=${env_var#REDIS_CONF_}
  key=$(echo "$redis_var" | cut -d= -f1 | tr '[:upper:]' '[:lower:]' | tr '_' '-')
  value=${redis_var#*=}
  echo "$key $value" >> "$REDIS_CONFIG_PATH"
done

if [ -f "$REDIS_CONFIG_PATH" ]; then
  exec redis-server "$REDIS_CONFIG_PATH"
else
  exec redis-server
fi

Docker above is built as a my-redis:8.2 and is run in cluster mode via docker-compose.yml:

version: "3.8"

services:
  redis: &redis
    image: my-redis:8.2
    volumes:
      - ./dev/docker-local/redis.conf:/redis.conf.tpl
    entrypoint:
      - bash
      - "-c"
      - "cp /redis.conf.tpl /etc/redis.conf && /entrypoint.sh"

  redis-pod-0:
    <<: *redis
    environment:
      REDIS_CONF_PORT: "6379"
    ports:
      - 6379:6379
      - 16379:16379
    networks:
      app_net:
        ipv4_address: 173.17.0.2

  redis-pod-1:
    <<: *redis
    environment:
      REDIS_CONF_PORT: "6380"
    ports:
      - 6380:6380
      - 16380:16380
    networks:
      app_net:
        ipv4_address: 173.17.0.3

  redis-pod-2:
    <<: *redis
    environment:
      REDIS_CONF_PORT: "6381"
    ports:
      - 6381:6381
      - 16381:16381
    networks:
      app_net:
        ipv4_address: 173.17.0.4

  setup-cluster:
    image: my-redis:8.2
    volumes:
      - ./dev/docker-local/setup_cluster.sh:/setup_cluster.sh
    entrypoint:
      - /setup_cluster.sh
    depends_on:
      - redis-pod-0
      - redis-pod-1
      - redis-pod-2
    networks:
      app_net:
        ipv4_address: 173.17.0.5

networks:
  app_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 173.17.0.0/16

where dev/docker-local/redis.conf is

cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-migration-barrier 1
cluster-config-file nodes.conf
appendonly yes
protected-mode no

and dev/docker-local/setup_cluster.sh:

#!/bin/bash

set -eu -o pipefail

declare -a CLUSTER_NODES=(
  "redis-pod-0:6379"
  "redis-pod-1:6380"
  "redis-pod-2:6381"
)

fail() {
  echo -e "\033[1;31m$1\033[0m" >&2
  exit 1
}

log() {
  echo -e "\033[1;34m$1\033[0m"
}

resolve_ip() {
  local hostname=$1
  getent hosts "$hostname" | awk '{ print $1 }'
}

get_resolved_cluster_nodes() {
  local host port
  for node in "${CLUSTER_NODES[@]}"; do
    host=$(cut -d: -f1 <<< "$node")
    port=$(cut -d: -f2 <<< "$node")
    echo "$(resolve_ip "$host"):$port"
  done
}

wait_for_service() {
  local host=$1
  local port=$2
  local timeout=$3
  for i in $(seq 0 $(( timeout-1 ))); do
    if nc -z "$host" "$port"; then
      log "$host:$port started in $i seconds."
      return
    fi
    log "Waiting for $host:$port to start..."
    sleep 1
  done
  fail "$host:$port did not start in $timeout seconds!"
}

apt-get update && apt-get install -y netcat-openbsd

for node in "${CLUSTER_NODES[@]}"; do
  wait_for_service "$(cut -d: -f1 <<< "$node")" "$(cut -d: -f2 <<< "$node")" 20
done

log "All cluster nodes have started. Creating Redis cluster..."

# shellcheck disable=SC2046
redis-cli --cluster create --cluster-replicas 0 $(get_resolved_cluster_nodes) --cluster-yes

echo
log "Redis cluster successfully created."

After running docker compose up, I checked whether everything was set properly:

redis-cli -h 127.0.0.1 -p 6379 CLUSTER NODES
a145e4b50dcc2b6549aaaee026aeed998b4de26e 173.17.0.4:6381@16381 master - 0 1754894023255 3 connected 10923-16383
83361f1e3d2c5e15e944075a01ce8acc090d9fca 173.17.0.3:6380@16380 master - 0 1754894022210 2 connected 5461-10922
9188de340dfd74e6110f2d45f65549a3751bd6f3 173.17.0.2:6379@16379 myself,master - 0 0 1 connected 0-5460

But this code does not work as expected:

import { createCluster } from 'redis';

const cluster = createCluster({
    rootNodes: [
        { url: 'redis://127.0.0.1:6379' },
        { url: 'redis://127.0.0.1:6380' },
        { url: 'redis://127.0.0.1:6381' },
    ],
    nodeAddressMap: {
        '173.17.0.2:6379': { host: '127.0.0.1', port: 6379 },
        '173.17.0.3:6380': { host: '127.0.0.1', port: 6380 },
        '173.17.0.4:6381': { host: '127.0.0.1', port: 6381 },
    },
    defaults: {
        socket: {
            connectTimeout: 5000,
            socketTimeout: 5000,
        },
    },
});

cluster.on('error', e => console.error('ERR', e));
cluster.on('ready', () => {
    console.log('READY');
    process.exit(0);
});

void (async (): Promise<void> => {
    await cluster.connect();
    console.log(await cluster.ping());
})();

It prints PONG, but after 5s it throws:
SocketTimeoutError: Socket timeout timeout. Expecting data, but didn't receive any in 5000ms.

The ready event is never fired.

ioredis works as expected with the same configuration. Am I doing anything wrong?

Node.js Version

v22.14.0

Redis Server Version

8.2.0

Node Redis Version

5.8.0

Platform

macOS

Logs

[nkaradzhov]

Hi @mdorda,

I haven't inspected all the configuration you provided, but given redis-cli -h 127.0.0.1 -p 6379 CLUSTER NODES looks good, I will assume you setup the cluster correctly.

You say

Cannot connect to cluster

But if you receive a PONG, you are actually connecting.

About the ready event, I think it is not fired in cluster/sentinel. There is an open issue for the sentinel. Im not sure if this is a bug or a feature request tbh. In any case, people find it natural to have a ready event in cluster/sentinel, so we might add it, will see.

Finally, you experience

SocketTimeoutError: Socket timeout timeout. Expecting data, but didn't receive any in 5000ms.

Which is perfectly normal, since you have socketTimeout: 5000 in your configuration object. Let me explain how socketTimeout works - if there is no data on the socket for N seconds, throw the error. The feature is intended to provide a way for users to cleanup very old connections. Socket timeout makes more sense in a standalone client, in cluster its more tricky because you have multiple connections and your workload could hit only certain connections ( because of sharding ), so some connections might not receive any data for a while. Any connection that doesn't receive data in those 5 seconds will error.

In your case you have a ping and then do nothing else. 5 seconds pass without any data in one of the sockets, so the error is thrown.

To summarize:
It looks like you can connect to the cluster normally, then do one single command and after 5 seconds you receive SocketTimeoutError, which is normal because there was no data on that socket for 5 seconds. I would suggest you remove the socketTimeout setting.

[mdorda]

Thank you so much for your quick response. Sorry about the misleading title. As I wrote at the very beginning:

“As far as I was able to debug it, it connects, downloads the cluster metadata and internal addresses, and then hangs indefinitely.”

You are right, I was able to connect. It all makes perfect sense now. It was a combination of misunderstanding how socketTimeout and events work. I thought it was hanging on some request and then timed out.

It seems to me that the documentation could be updated a bit. Anyway, thank you again, everything works as expected now!