When working in server mode with remote_storage: SFTP, ssh sessions not closed

[No0key]

Hello!
I'm running clickhouse-backup as systemd service (clickhouse-backup -c /path/to/my-config server) and when i'm executing any API call(list,upload,download) that interacts with remote SFTP server, it does not close SSH sessions.

Backup host

ssh -V
OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020

clickhouse-backup --version
Version:	 2.0.0
Git Commit:	 93f6c2f168b63db4ab7c8cd51fcc317884752b84
Build Date:	 2022-09-04

lsb_release -a

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

SFTP Server

ssh -V
OpenSSH_7.6p1 Ubuntu-4ubuntu0.5, OpenSSL 1.0.2n  7 Dec 2017

lsb_release -a 
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

I've enabled on remote SFTP server debug and below is output.

CORRECT
clickhouse-backup -c /path/to/my-config list

Oct 19 08:19:04 server_name systemd[1]: Started Session 285861 of user backupuser.
Oct 19 08:19:04 server_name sftp-server[71125]: session opened for local user backupuser from [backup_host_ip]
Oct 19 08:19:04 server_name sftp-server[71125]: received client version 3
Oct 19 08:19:04 server_name sftp-server[71125]: debug3: request 1: opendir
Oct 19 08:19:04 server_name sftp-server[71125]: opendir "/path/to/backups/directory/"
Oct 19 08:19:04 server_name sftp-server[71125]: debug1: request 1: sent handle handle 0
Oct 19 08:19:04 server_name sftp-server[71125]: debug1: request 2: readdir "/path/to/backups/directory/" (handle 0)
Oct 19 08:19:04 server_name sftp-server[71125]: debug1: request 2: sent names count 100
Oct 19 08:19:04 server_name sftp-server[71125]: debug1: request 3: readdir "/path/to/backups/directory/" (handle 0)
Oct 19 08:19:04 server_name sftp-server[71125]: debug1: request 3: sent names count 94
Oct 19 08:19:04 server_name sftp-server[71125]: debug1: request 4: readdir "/path/to/backups/directory/" (handle 0)
Oct 19 08:19:04 server_name sftp-server[71125]: debug3: request 4: sent status 1
Oct 19 08:19:04 server_name sftp-server[71125]: sent status End of file
Oct 19 08:19:04 server_name sftp-server[71125]: debug3: request 5: close handle 0
Oct 19 08:19:04 server_name sftp-server[71125]: closedir "/path/to/backups/directory/"
Oct 19 08:19:04 server_name sftp-server[71125]: debug3: request 5: sent status 0
Oct 19 08:19:04 server_name sftp-server[71125]: sent status Success
Oct 19 08:19:14 server_name sftp-server[71125]: debug1: read eof
Oct 19 08:19:14 server_name sftp-server[71125]: session closed for local user backupuser from [backup_host_ip]

NOT CORRECT
curl localhost:7171/backup/list

Oct 19 08:08:04 server_name systemd[1]: Started Session 282897 of user backupuser.
Oct 19 08:08:04 server_name sftp-server[70423]: session opened for local user backupuser from [backup_host_ip]
Oct 19 08:08:04 server_name sftp-server[70423]: received client version 3
Oct 19 08:08:04 server_name sftp-server[70423]: debug3: request 1: opendir
Oct 19 08:08:04 server_name sftp-server[70423]: opendir "/path/to/backups/directory/"
Oct 19 08:08:04 server_name sftp-server[70423]: debug1: request 1: sent handle handle 0
Oct 19 08:08:04 server_name sftp-server[70423]: debug1: request 2: readdir "/path/to/backups/directory/" (handle 0)
Oct 19 08:08:04 server_name sftp-server[70423]: debug1: request 2: sent names count 100
Oct 19 08:08:04 server_name sftp-server[70423]: debug1: request 3: readdir "/path/to/backups/directory/" (handle 0)
Oct 19 08:08:04 server_name sftp-server[70423]: debug1: request 3: sent names count 94
Oct 19 08:08:04 server_name sftp-server[70423]: debug1: request 4: readdir "/path/to/backups/directory/" (handle 0)
Oct 19 08:08:04 server_name sftp-server[70423]: debug3: request 4: sent status 1
Oct 19 08:08:04 server_name sftp-server[70423]: sent status End of file
Oct 19 08:08:04 server_name sftp-server[70423]: debug3: request 5: close handle 0
Oct 19 08:08:04 server_name sftp-server[70423]: closedir "/path/to/backups/directory/"
Oct 19 08:08:04 server_name sftp-server[70423]: debug3: request 5: sent status 0
Oct 19 08:08:04 server_name sftp-server[70423]: sent status Success

/var/log/syslog
Oct 18 16:22:03 server_name kernel: [12161367.735233] cgroup: fork rejected by pids controller in /user.slice/user-1000.slice/session-281775.scope

ps aux | grep ssh 
...
backupu+   69687  0.0  0.0 101704  4388 ?        S<   07:55   0:00 sshd: backupu@notty
root       70008  0.0  0.0 101564  7104 ?        S<s  08:00   0:00 sshd: backupu [priv]
backupu+   70012  0.0  0.0 101704  4580 ?        S<   08:00   0:00 sshd: backupu@notty
root       70417  0.0  0.0 101564  7128 ?        S<s  08:08   0:00 sshd: backupu [priv]
backupu+   70421  0.0  0.0 101704  4376 ?        S<   08:08   0:00 sshd: backupu@notty
...

ps aux | grep ssh | wc -l
5531

So this behaviour results reaching maximum pids for SSHD service on SFTP server. SSH proceses spawned but not closed and when it reaches maximum, then in clickhouse-backup logs we see
'error': "can't connect to SFTP: ssh: unexpected packet in response to channel open: <nil>"

[Slach]

oh, ok. thanks for reporting