DOS container does not close connections results in unclosed file descriptors

[agoscinski]

In Python 3.13 open connections that are not explicitly closed do not release their file descriptor anymore. I am not sure where this change comes from. The only thing I could find is that now a ResourceWarning is emitted when a connection is not closed (see https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection). This could indicate that the code keeps now track of these resources which avoids a garbage collection that would close the file descriptor but this is just a guess.

In the test suite, any test initializing storage components, which is almost every test, file descriptors are open and kept open. This results that at the end of the test suite we have up to 5000 file descriptors open which makes the ssh proxy test fail since it needs to open another file descriptor which it cannot (see error in CI). I find this also very weird, since sqlalchemy can open further file descriptor through connections but not select. Also if I check with ulimit -n, the limit of file descriptor has not been reached yet. Maybe because we in the ´sqlalchemy` case we open the same file at different positions this is considered differently.

The main source of open connections seems to be the disk object storage container. With each instantiation we create a new object with new connections that are never explicitly closed. I think the fix is to reuse just the open connection once the container has been created. In PR #10220 I am playing around to make the Container a singleton, but I have not found the right solution yet.

I must say the code that open and closes connections is quite intransparent what should close the connection when it is open. There are public functions that open connections but do not close them, basically trusting the user of the function to close it without even specifying in documentation. Then there are some enforcements of freeing the connections enforcing garbage collection, which I think need to be there because we dispose the engine before we close connections. but if we would first close the connection and then dispose the engine it should be not needed. All this makes it time consuming to fix this issue. Are there any design documents available? I still not fully understand the responsibility of resources between the migrator, backend and manager.