Too many connections

[toshski]

This change aims to address issues related to "Too Many Connections" errors, particularly for users experiencing such problems due to the utilization of numerous Custom Sites.

The modification introduces a new database function called GetCommonDB, mirroring the existing GetDB function. However, it establishes and maintains a shared connection pool created at startup, which should remain open throughout. In the case of MySQL, the gorm.Open function invoked within these functions initiates a connection pool, not a single connection.

The db.GetDB functions utilized during scraping and some update functions in the models have been updated to use the newly introduced GetCommonDB. By default, XBVR will still attempt to create as many connections as requested, potentially resulting in the same "Too Many Connections" failures. However, the size of the common connection pool can optionally be restricted by configuring the runtime parameter dB_connection_pool_size or the environment variable DB_CONNECTION_POOL_SIZE. This limitation then regulates the number of concurrent connections the GetCommonDB pool will open. Queries surpassing this limit will be queued until a connection becomes available, after which they will proceed.

Setting the pool size too low may decelerate scraping, as threads wait for an idle connection if the limit is exceeded. Conversely, setting it excessively high may still lead to "Too Many Connections" errors, as other functions using GetDB still need to create other connections.

To enhance clarity, the db variable, normally assigned to gorm.DB, has been renamed to commonDB when GetCommonDB is used. This adjustment helps developers easily discern which connection pool is in use. It is imperative that developers refrain from calling Close on the common pool.

Tests have indicated that utilizing a common pool yields faster results. For instance, performing 10,000 reads of a scene, pre-loading tags, cast, files, history, and cue points takes approximately 40 seconds with GetCommonDB compared to about 60 seconds with GetDB.