Working with database

Default Rails config for database uses value of RAILS_MAX_THREADS ENV variable to set database connections pool size. Fallback value is 5.

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

development:
  <<: *default
  database: example_development

test:
  <<: *default
  database: example_test

production:
  <<: *default
  url: <%= ENV['DATABASE_URL'] %>

It is a common situation when there are more threads running in worker process. When active threads number is bigger than pool size, ActiveRecord fails to make extra connection.

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5 seconds (waited 5.000166496 seconds). The max pool size is currently 5; consider increasing it.

There are few ways to mitigate this problem:

1. [Simple] Set pool size from another ENV variable with fallback to RAILS_MAX_THREADS

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("DB_POOL_SIZE") { ENV.fetch("RAILS_MAX_THREADS") { 5 } } %>

2. [Best] Set pool size to be equal to total threads count and drop connections of hypervisor process

# in /config/initializers/advanced_sneakers_activejob.rb
AdvancedSneakersActiveJob.configure do |config|
  config.sneakers = {
    hooks: {
      before_fork: lambda {
        Rails.logger.info('Worker: Disconnecting from the database')
        ActiveRecord::Base.connection_pool.disconnect!
      },
      after_fork: lambda {
        Rails.logger.info('Worker: Reconnecting to the database')
        threads_count = Sneakers::Worker::Classes.call.map { |w| w.queue_opts.fetch(:threads) { AdvancedSneakersActiveJob.config.sneakers[:threads] } }.sum
        config = ActiveRecord::Base.configurations[Rails.env] || Rails.application.config.database_configuration[Rails.env]
        config['pool'] = threads_count
        ActiveRecord::Base.establish_connection(config)
        Rails.logger.info("Worker: Reconnected to the database with pool size #{threads_count}")
      }
    }
  }
end

3. [Tradeoff] If the database connection pool size is limited (e.g. by price for DB-service plan), then worker classes should be forced to use the pool. Take into account, that it is a bottleneck.

class MyAwesomeJob < ApplicationJob
  queue_as :cleanup

  def perform(params)
    ActiveRecord::Base.connection_pool.with_connection do
      # Do some database-related job
    end
  end
end