Admin Interface Port Reuse with Hot Reload

[bspavan25]

Title: Admin Interface Port Reuse with Hot Reload

Description:
I'm using the hot restart Python wrapper provided by Envoy (https://www.envoyproxy.io/docs/envoy/latest/operations/hot_restarter) to run Envoy with a configuration that includes an admin interface. When I include the following admin interface configuration in my Envoy config:

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 8001

However, I encounter a "address already in use" error when trying to hot reload:

[2025-01-22 21:19:43.436][10832][critical][main] [source/server/server.cc:412] error initializing config '  bin/config.yaml': cannot bind '127.0.0.1:8001': Address already in use

However, when I add the following socket options to the admin interface configuration:

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 8001
  socket_options:
    - level: 1  # SOL_SOCKET
      name: 15  # SO_REUSEPORT
      int_value: 1
      state: STATE_PREBIND

The hot reload works as expected.

My understanding is that the admin interface runs on the main thread and does not allow port reuse by default. The added socket options enable port reuse, allowing the hot reload to work. I would like to confirm if this understanding is correct and if this approach is recommended. Are there any potential issues with using this socket option for the admin interface?

[Relevant Links:]
https://www.envoyproxy.io/docs/envoy/latest/operations/hot_restarter
https://github.com/envoyproxy/envoy/blob/41d8ccc79c4b98104475b55b5639567b14466190//restarter/hot-restarter.py

[adisuissa]

At a high-level, your description seems correct, and all listeners on ports that are to be reused should be defined as such.

cc @ravenblackx who may be using hot-restart and might have an answer (forgive me in advance if you are not using hot-restart, just let me know and I'll try to find someone who does).

[KBaichoo]

We use hot restart with an in-house equivalent to the restart_wrapper but haven't yet ran into this issue.

I wonder if you had multiple more than 1 envoy hot restart going on (3 total envoys)?

Hot restart works by establishing UDS sockets b/w the parent and child envoy process and sending an admin shutdown to the parent before the child admin takes over. src. It's also possible that your issue is due to TIME-Wait set on your machine for the given socket.

[ravenblackx]

Yeah, we are also using hot-restart and haven't run into "address already in use" in this form. I believe that the hot restart operation closes the admin socket on the old instance before opening the admin socket on the new instance, so my guess would be that the hot restart script as used is not correctly setting up the parent/child relationship between the two instances so that they communicate as intended. (Though KBaichoo's suggestion of maybe two or more restarts happening at once, or two instances running with the same port configured that aren't connected by hot restart, are also possibilities.)