FreeBSD/OSD.cc: add client_messenger to the avoid_ports set.

Observed "feature":
  During rebind due to a "wrongly marked down" log message, FreeBSD is
  able to bind to the port used by client_messenger.
  The Linux variant avoids that port because it is already in use.

Result:
  In FreeBSD there would be 2 listeners on the port, and due to the rebind
  they have different nonces. (This is written in the logfile)
  But they also will expect different protocols on that same port.

This is likely due to an interpretation difference in the SO_REUSEADDR
socket option:

Linux:
  SO_REUSEADDR
        Indicates that the rules used in validating addresses supplied
        in a bind(2)  call  should  allow  reuse  of  local addresses.
        For AF_INET sockets this means that a socket may bind,
        except when there is an active listening socket bound to the address.
        When the listening socket is bound to INADDR_ANY with a specific port
        then it is not possible to bind to this port for any local address.
        Argument is an integer boolean flag.

FreeBSD:
  SO_REUSEADDR
        Enables local address reuse
        indicates that the rules used in validating addresses supplied in a
        bind(2) system call should allow reuse of local addresses.

So FreeBSD doesn't guarantee that the connection is refused when there is already
a connection. So it is best avoided during rebinding otherwise any of the
cluster_messengers will attach to the port.

FreeBSD log with wrong connect:
bb98d80  0 log_channel(cluster) log [WRN] : map e18 wrongly marked me down
bb98d80  1 -- 127.0.0.1:6801/17881 rebind rebind avoid 6801,6802,6803
bb98d80  1 -- 127.0.0.1:6801/17881 shutdown_connections
bb98d80  1 -- 127.0.0.1:6800/1017881 _finish_bind bind my_inst.addr is 127.0.0.1:6800/1017881
bb98d80  1  Processor -- start
bb98d80  1 -- 127.0.0.1:6802/17881 rebind rebind avoid 6801,6802,6803
bb98d80  1 -- 127.0.0.1:6802/17881 shutdown_connections
bb98d80  1 -- 127.0.0.1:0/17881 learned_addr learned my addr 127.0.0.1:0/17881
bb98d80  1 -- 127.0.0.1:6804/1017881 _finish_bind bind my_inst.addr is 127.0.0.1:6804/1017881

FreeBSD with the correct behaviour:
bb98d80  0 log_channel(cluster) log [WRN] : map e17 wrongly marked me down
bb98d80  1 -- 127.0.0.1:6802/15296 rebind rebind avoid 6801,6802,6803,6812
bb98d80  1 -- 127.0.0.1:6802/15296 shutdown_connections
bb98d80  1 -- 127.0.0.1:6806/1015296 _finish_bind bind my_inst.addr is 127.0.0.1:6806/1015296
bb98d80  1  Processor -- start
bb98d80  1 -- 127.0.0.1:6803/15296 rebind rebind avoid 6801,6802,6803,6812
bb98d80  1 -- 127.0.0.1:6803/15296 shutdown_connections
bb98d80  1 -- 127.0.0.1:0/15296 learned_addr learned my addr 127.0.0.1:0/15296
bb98d80  1 -- 127.0.0.1:6807/1015296 _finish_bind bind my_inst.addr is 127.0.0.1:6807/1015296

Signed-off-by: Willem Jan Withagen <[email protected]>

src/osd/OSD.cc

@@ -7120,6 +7120,12 @@ void OSD::_committed_osd_maps(epoch_t first, epoch_t last, MOSDMap *m)
 	start_waiting_for_healthy();
 
 	set<int> avoid_ports;
+#if defined(__FreeBSD__)
+        // prevent FreeBSD from grabbing the client_messenger port during
+        // rebinding. In which case a cluster_meesneger will connect also 
+	// to the same port
+	avoid_ports.insert(client_messenger->get_myaddr().get_port());
+#endif
 	avoid_ports.insert(cluster_messenger->get_myaddr().get_port());
 	avoid_ports.insert(hb_back_server_messenger->get_myaddr().get_port());
 	avoid_ports.insert(hb_front_server_messenger->get_myaddr().get_port());