[Bug] stunnel process is not killed after unmount

[leakingtapan]

There are numerous stunnel processes left after mount point is unmounted for many times.
This causes process leak and port leak.

The following is htop output of the leaking stunnel process:

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    1 root       20   0  113M 14064 10172 S  0.0  0.2  0:01.27 /bin/aws-efs-csi-driver --endpoint=unix:/csi/csi.sock --logtostderr --v=5
  103 root       20   0  111M  5684  4824 S  0.0  0.1  0:00.04 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.49697c3c-123e-11ea-84e4-02e886441bde.volumes.k
   96 root       20   0  111M  5732  4872 S  0.0  0.1  0:00.22 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.49697c3c-123e-11ea-84e4-02e886441bde.volumes.k
   82 root       20   0  111M  5940  5080 S  0.0  0.1  0:00.03 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.c836f47c-123d-11ea-bb3d-0a95942502dc.volumes.k
   75 root       20   0  111M  6064  5204 S  0.0  0.1  0:00.04 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.c836f47c-123d-11ea-bb3d-0a95942502dc.volumes.k
   63 root       20   0  111M  6152  5292 S  0.0  0.1  0:00.03 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.cc6f0b25-123c-11ea-84e4-02e886441bde.volumes.k
   56 root       20   0  111M  5920  5060 S  0.0  0.1  0:00.04 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.cc6f0b25-123c-11ea-84e4-02e886441bde.volumes.k
   28 root       20   0  111M  6164  5300 S  0.0  0.1  0:00.03 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.4bf0b8f8-123c-11ea-84e4-02e886441bde.volumes.k
   20 root       20   0  111M  6096  5236 S  0.0  0.1  0:00.03 `- stunnel /var/run/efs/stunnel-config.fs-e8a95a42.var.lib.kubelet.pods.4bf0b8f8-123c-11ea-84e4-02e886441bde.volumes.k
  129 root       20   0 11752  3072  2720 S  0.0  0.0  0:00.00 bash
  135 root       20   0 17128  3712  2744 R  0.0  0.0  0:00.03 `- htop

linux distro: amazonlinux:2 container image
kernel version: 4.14.133
efs-utils version: 1.9

[maxbecke]

Thank you for the bug report! Could you provide some more information such as what linux distribution you are using, the kernel version, and and the efs-utils version?

[leakingtapan]

@maxbecke

linux distro: amazonlinux:2 container image
kernel version: 4.14.133
efs-utils version: 1.9

[leakingtapan]

I think the problem is when efs is mounted using efs mount helper a child stunnel process is spawned then tls mount option is used. However, when unmont is called to unmount the filesystem, it doesn't know the existence of stunnel process (since unmount is generic command), so stunnel process is never killed. Also, although, this issue happens for me in the container environment, it applies to non-container case as well.

What's efs's recommendation on unmount a filesystem with tls mount option?

[zhanyuya]

Thanks for your feedback.

Stunnel isn't closed right away after umount. Please understand efs-utils requires 30s (by default) grace period to close Stunnel.

If the 30s waiting time is too long for you, please follow configuration below,

1. Edit /etc/amazon/efs/efs-utils.conf
2. Find unmount_grace_period_sec in [mount-watchdog] section
3. Change the number to the desired one (in second)

[leakingtapan]

Thx for the input @zhanyuya
Since we are running efs mount helper in a containerized environment where systemd is not favored, what’s the recommended approach to start watch dog?

[Cappuccinuo]

This issue has been solved after kubernetes-sigs/aws-efs-csi-driver#113 this pull request is merged in csi-driver, which enable watchdog to be launched so that stunnel can be properly killed.

[couling]

I'm having trouble understanding the context of why this was closed. This bug seems to have been raised against efs-utils but then closed because of a "fix" / work around in Kubernetes (only). So what happens for those of us who are not using Kubernetes?

Did I miss something obvious?