Lots of CLOSE_WAITS to Vault on petems-hiera_vault v2.0.0

[impisuperstar]

Hi,

Version:
puppetserver version: 6.8.0
Puppet v6.22.1
petems-hiera_vault v2.0.0

Puppet agent:
Puppet v6.13.0

Vault 1.12.0

Haproxy 2.1.4

After the installing hiera_vault we begin to see a lot of CLOSE_WAITS, it looks like application does not close the connection properly.

#25 - we found the previous issue about it, but the problem begin to reproduce on the latest version also.

Puppet server does not send fin package from it side:

16:56:43.070098 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [S], seq 3899952215, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
16:56:43.070120 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [S.], seq 1096888154, ack 3899952216, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
16:56:43.070135 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 1, win 342, length 0
16:56:43.109275 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 1:227, ack 1, win 342, length 226
16:56:43.109296 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 227, win 350, length 0
16:56:43.150004 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 1:1950, ack 227, win 350, length 1949
16:56:43.150011 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 1950, win 1365, length 0
16:56:43.206679 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 227:309, ack 1950, win 1365, length 82
16:56:43.206689 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 309, win 350, length 0
16:56:43.206814 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 309:315, ack 1950, win 1365, length 6
16:56:43.206820 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 315, win 350, length 0
16:56:43.207070 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 315:384, ack 1950, win 1365, length 69
16:56:43.207077 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [.], ack 384, win 350, length 0
16:56:43.207638 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 1950:2025, ack 384, win 350, length 75
16:56:43.207646 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 2025, win 1365, length 0
16:56:43.211176 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 384:837, ack 2025, win 1365, length 453
16:56:43.212136 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 2025:2574, ack 837, win 359, length 549
16:56:43.219959 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [P.], seq 837:1338, ack 2574, win 1396, length 501
16:56:43.223363 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [P.], seq 2574:2883, ack 1338, win 367, length 309
16:56:43.262512 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 2883, win 1426, length 0

vault try to close connection from it side and send tcp fin:
17:01:43.224629 IP 127.0.0.1.5000 > 127.0.0.1.48228: Flags [F.], seq 2883:2936, ack 1338, win 367, length 53
17:01:43.264539 IP 127.0.0.1.48228 > 127.0.0.1.5000: Flags [.], ack 2937, win 1426, length 0

and number of close_wait begin to increase from time to time:
CLOSE_WAIT
netstat -ntulpa | grep CLOSE_WAIT
tcp 0 0 x.x.x.1:33124 x.x.x.2:8200 CLOSE_WAIT 36977/haproxy
tcp 54 0 127.0.0.1:48228 127.0.0.1:5000 CLOSE_WAIT 42943/java

hiera config:

• name: Hiera-vault lookup
  lookup_key: hiera_vault
  options:
  confine_to_keys:
  - '^.*password$'
  ssl_verify: false
  ssl_ca_cert: <path_to_cert>
  address: <haproxy address, which balance and forward traffic to Vault>
  token:
  default_field: secret
  mounts:
  puppet:
  - ''

[StanvanHoorn]

Hi @petems,

We are also running into this issue.

Did you have any change to take a look at this?

Thanks

[StanvanHoorn]

Hi @petems,

Do you have any plans to take a look at this?

Thanks

[StanvanHoorn]

@impisuperstar Did you find a fix or a solution for it?

[markri]

Following coud serve as a temporary work-around. Add a crontab entry like this (replace the 1.1.1.1 with your vault instance IP).

*/10 * * * * ss --tcp state CLOSE-WAIT 'dst 1.1.1.1' --kill

I really hate to come up with a solution like this, but my poor Ruby skills forced me to be creative until something more sustainable emerges.