Proxysql 2.5 and ssl to mysql server never close fd on authority file.

[quetzal]

Hello,

I've identified an issue with proxysql. It's present on proxysql 2.5.1 from percona repository and 2.5.3 from proxysql .deb on Ubuntu focal (not tested other versions / other os). It's not present on 2.4.8 from percona repository.

I use proxysql with a ssl connection to the mysql servers. I configure it via ansible.
I configure a ssl CA, cert and key. For the ca:

• name: Configure ssl ca
  proxysql_global_variables:
  login_user: '{{ login_user }}'
  login_password: '{{ login_password }}'
  variable: 'mysql-ssl_p2s_ca'
  value: '/etc/ssl/mysql-ca.pem'

Where /etc/ssl/mysql-ca.pem is a file I copy before (same for key and pem).
I configure the cypher:

• name: Configure ssl cypher
  proxysql_global_variables:
  login_user: '{{ login_user }}'
  login_password: '{{ login_password }}'
  variable: 'mysql-ssl_p2s_cipher'
  value: 'ECDHE-RSA-AES256-SHA'

With proxysql 2.5.X, the file descriptor on '/etc/ssl/mysql-ca.pem' is never closed and seem to be opened at each new connection and never closed. So the file descriptor number grow until the limit is reached (I limit it to LimitNOFILE=204800) and then proxysql crash and restart (because it can't open new file descriptors).

On proxysql 2.4.8, i don't see any file descriptor opened on this file (so i assume it's cached on startup), and my file descriptor opened number still between 80 and 100 (when it reach 204800 after few days on proxysql 2.5.3).

It seem easy to reproduce (use proxysql 2.5 with mysql servers with ssl and a big number of connections).

Thank you!

[renecannao]

Thank you for the report.

In ProxySQL 2.5.1 we upgraded OpenSSL . Not pointing finger yet, but because ProxySQL doesn't open the file directly (it just sets the option to use that file), this is probably an OpenSSL bug.
We will investigate

[jhampson-dbre]

We seem to be having similar issue:

sudo lsof -p 6354 | grep proxysql-ca | wc -l
25695

This number rises by a few thousand each day until proxysql crashes. This number seems to roughly match the combined number of connOK from stats_mysql_connection_pool for each hostgroup with use_ssl enabled.

I noticed in release 2.5.1 CA file caching was introduced:

Added x509 shared cache improving backend MySQL connections creation speed for large CA files #4120

It looks to me like the mariadb client library was patched with an explicit open on the CAfile to implement the caching:

proxysql/deps/mariadb-client-library/openssl.c.x509cache.patch

Line 33 in c640f1b

+ fd = open(CAfile, O_RDONLY);

and I don't see where this ever gets closed. But it's been a long time since I've written anything in C so maybe it is being closed elsewhere and I just missed it. I thought it would be worth pointing out just in case it ends up being helpful for resolving this issue. Thanks!

[renecannao]

Thanks @jhampson-dbre for pointing out the root cause.
PR is already available