Allow GoodJob::Job#retry_job's transaction to be joinable (fix CI for current rails-head)

[Earlopain]

It is now >= 3.2 only

[Earlopain]

Locks like the failures are legit, I can reliably reproduce with rails-head locally.

Failures:

  1) GoodJob::Job#retry_job when job is retried updates the original job
     Failure/Error: expect(POSTGRES_NOTICES).to be_empty
       expected `["Connection returned to pool with 1 advisory locks", "Connection returned to pool with 1 advisory locks"].empty?` to be truthy, got false
     # ./spec/support/postgres_notices.rb:34:in `block (2 levels) in <main>'
     # ./spec/support/logger.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'

  2) GoodJob::Job#retry_job when job is retried when run inline executes the job
     Failure/Error: expect(POSTGRES_NOTICES).to be_empty
       expected `["Connection returned to pool with 1 advisory locks", "Connection returned to pool with 1 advisory locks"].empty?` to be truthy, got false
     # ./spec/support/postgres_notices.rb:34:in `block (2 levels) in <main>'
     # ./spec/support/logger.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'

  3) GoodJob::Job#retry_job when job arguments cannot be deserialized ignores the error and discards the job
     Failure/Error: expect(POSTGRES_NOTICES).to be_empty
       expected `["Connection returned to pool with 1 advisory locks", "Connection returned to pool with 1 advisory locks"].empty?` to be truthy, got false
     # ./spec/support/postgres_notices.rb:34:in `block (2 levels) in <main>'
     # ./spec/support/logger.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'

Finished in 2 minutes 15.5 seconds (files took 1.81 seconds to load)
567 examples, 3 failures

Failed examples:

rspec ./spec/app/models/good_job/job_spec.rb:135 # GoodJob::Job#retry_job when job is retried updates the original job
rspec ./spec/app/models/good_job/job_spec.rb:159 # GoodJob::Job#retry_job when job is retried when run inline executes the job
rspec ./spec/app/models/good_job/job_spec.rb:210 # GoodJob::Job#retry_job when job arguments cannot be deserialized ignores the error and discards the job

Will spend some time bisecting later

[Earlopain]

Caused by rails/rails#53118. There's rails/rails#53147 (which probably is something different, since the PR that caused this was a proposed fix for that kind of problem). Will try to reduce into something reportable

[bensheldon]

@Earlopain thank you so much for looking into this!

It is interesting that it looks like it's just GoodJob::Job#retry_job that has problems. I could imagine that GoodJob itself is doing something unexpected (or wrong!) because GoodJob jumps through a lot of hoops in #retry_job to simulate an Active Job retry.

From reading the Rails change, it looks like it heavily enforces "if there is not an open transaction, the current database connection is checked in" when a Rails executor completes. This should not a problem because nested Rails Executors should be noops, and it's only the outermost executor that should run the callbacks (I'm 90% sure of this; if I am wrong that's a problem!) and by the outermost executor GoodJob should have released all it's advisory locks.

[Earlopain]

I believe your assumption is correct. Here are my findings:

• The spec helper calls executor.perform which does execute each time (I think):
  

  good_job/spec/support/rails_executor.rb

  Lines 17 to 21 in 62af353

  	Rails.application.executor.perform do
  	ActiveRecord::Base.connection.disable_query_cache!
  	example.run
  	end
  	else

  
  https://github.com/rails/rails/blob/b050366a9f680aea186110c7fb5b867e5e8f5020/activesupport/lib/active_support/execution_wrapper.rb#L100-L108
• wrap on the other hand checks if it is already active: https://github.com/rails/rails/blob/b050366a9f680aea186110c7fb5b867e5e8f5020/activesupport/lib/active_support/execution_wrapper.rb#L86-L98
• Switching the call to wrap does make all tests pass. However the distinction between perform and wrap in the spec helper is very deliberate.

Rails own executor_around_test_case config is documented as using wrap but uses perform instead. I think that's just a docs error though, the caveat from rails/rails#43550 is still there. I'll just open a PR for that to fix.

Does is seem reasonable for you to switch to wrap instead of perform? Seems a bit weird to me and I don't understand it enough anyways. Maybe its just an order-of-execution thing since the rails PR did a new hook. But even if, I wouldn't know how to fix. It would be nice if you could take this over when you got some time.

[bensheldon]

hmmmmm.

By the time any test gets to good_job/spec/support/rails_executor.rb, there should not be any Advisory Locks so I believe that executor#perform is the correct one to use here. If there are still advisory locks remaining, that seems like GoodJob (or the test) has a problem.

It does seem interesting that it is just those tests, which makes me suspect that maybe this is GoodJob's problem.

I would be curious about the callers stacktrace when the connection is checked back in:

good_job/spec/support/postgres_notices.rb

Line 23 in 62af353

warning = "Connection returned to pool with #{count} advisory locks"

...cause I am unsure whether the Notice is being triggered at the very end of the test run (when I do expect all advisory locks to be released), or somewhere in the middle of the test run (when there would be open advisory locks and lots of Rails Executor boundaries being being cross that I believe should be noops).

Also, reading the the test file, this is sorta curious. Now why did I do that?!:

good_job/spec/app/models/good_job/job_spec.rb

Lines 353 to 355 in 62af353

	around do |example|
	Rails.application.executor.wrap { example.run }
	end

[Earlopain]

Here are caller locations:

1

["/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `block in make_lambda'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:178:in `block in call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `catch'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `block in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:179:in `call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `block in invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `each'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:108:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:912:in `_run_checkin_callbacks'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:579:in `block (2 levels) in checkin'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:576:in `block in checkin'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:575:in `checkin'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:392:in `release_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:197:in `block in complete'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_handler.rb:107:in `block (2 levels) in each_connection_pool'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:31:in `each_value'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:31:in `block in each_pool_config'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:30:in `each_value'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:30:in `each_pool_config'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_handler.rb:106:in `block in each_connection_pool'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/map.rb:265:in `block in each_value'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:101:in `block in each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:100:in `each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:100:in `each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/map.rb:276:in `each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/map.rb:265:in `each_value'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_handler.rb:105:in `each_connection_pool'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `complete'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:36:in `before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:384:in `block in make_lambda'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:178:in `block in call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `catch'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `block in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:179:in `call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `block in invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `each'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:108:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:142:in `complete'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:136:in `complete!'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:96:in `wrap'",
 "/home/earlopain/Documents/good_job/lib/good_job/adapter.rb:140:in `enqueue_at'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueuing.rb:131:in `raw_enqueue'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueue_after_transaction_commit.rb:24:in `raw_enqueue'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueuing.rb:118:in `block in enqueue'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:120:in `block in run_callbacks'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:40:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:39:in `instrument'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/railties/job_runtime.rb:18:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:21:in `block (2 levels) in <module:Instrumentation>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `block in run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/tagged_logging.rb:143:in `block in tagged'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/tagged_logging.rb:38:in `tagged'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/tagged_logging.rb:143:in `tagged'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/broadcast_logger.rb:241:in `method_missing'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/logging.rb:39:in `tag_logger'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/logging.rb:28:in `block (2 levels) in <module:Logging>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `block in run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:140:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueuing.rb:117:in `enqueue'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/exceptions.rb:153:in `block in retry_job'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:40:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:39:in `instrument'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/railties/job_runtime.rb:18:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/exceptions.rb:152:in `retry_job'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:483:in `block (3 levels) in retry_job'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:626:in `block in within_new_transaction'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:623:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:367:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in `transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:234:in `block in transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:414:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_handling.rb:310:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:233:in `transaction'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:482:in `block (2 levels) in retry_job'",
 "/home/earlopain/Documents/good_job/lib/good_job/current_thread.rb:113:in `within'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:478:in `block in retry_job'",
 "/home/earlopain/Documents/good_job/app/models/concerns/good_job/advisory_lockable.rb:368:in `with_advisory_lock'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:457:in `retry_job'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:138:in `block (5 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:395:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:223:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:193:in `matches?'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:51:in `block in handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:48:in `handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:65:in `to'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:139:in `to'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:136:in `block (4 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-rails-6.1.3/lib/rspec/rails/adapters.rb:75:in `block (2 levels) in <module:MinitestLifecycleAdapter>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_rails_queue_adapter.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_good_job.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:19:in `block (3 levels) in <main>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:104:in `perform'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/postgres_notices.rb:35:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/logger.rb:7:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:259:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:646:in `block in run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:607:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2091:in `with_suite_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:116:in `block in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:115:in `run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `kernel_load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:23:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:455:in `exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:35:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:29:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:28:in `block in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:20:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `<main>'"]

2

["/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `block in make_lambda'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:178:in `block in call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `catch'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `block in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:179:in `call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `block in invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `each'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:108:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:912:in `_run_checkin_callbacks'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:579:in `block (2 levels) in checkin'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:576:in `block in checkin'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:575:in `checkin'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:392:in `release_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_handling.rb:310:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:410:in `with_transaction_returning_status'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:366:in `save!'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/suppressor.rb:56:in `save!'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:485:in `block (3 levels) in retry_job'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:626:in `block in within_new_transaction'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:623:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:367:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in `transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:234:in `block in transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:414:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_handling.rb:310:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:233:in `transaction'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:482:in `block (2 levels) in retry_job'",
 "/home/earlopain/Documents/good_job/lib/good_job/current_thread.rb:113:in `within'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:478:in `block in retry_job'",
 "/home/earlopain/Documents/good_job/app/models/concerns/good_job/advisory_lockable.rb:368:in `with_advisory_lock'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:457:in `retry_job'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:138:in `block (5 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:395:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:223:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:193:in `matches?'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:51:in `block in handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:48:in `handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:65:in `to'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:139:in `to'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:136:in `block (4 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-rails-6.1.3/lib/rspec/rails/adapters.rb:75:in `block (2 levels) in <module:MinitestLifecycleAdapter>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_rails_queue_adapter.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_good_job.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:19:in `block (3 levels) in <main>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:104:in `perform'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/postgres_notices.rb:35:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/logger.rb:7:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:259:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:646:in `block in run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:607:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2091:in `with_suite_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:116:in `block in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:115:in `run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `kernel_load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:23:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:455:in `exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:35:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:29:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:28:in `block in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:20:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `<main>'"]

Previously there where 9 checkins, now there are 11, which checks out since there are 2 warnings. I briefly looked at these already but couldn't make sense of them. It is too deep into internals for me.

good_job/app/models/good_job/job.rb

Line 481 in 62af353

self.class.transaction(joinable: false, requires_new: true) do

This uses joinable: false which seems relevant. If you add some pp debugging around it:

"BEFORE TRANSACTION"
"INSIDE TRANSACTION BEFORE AJ.retry_job"
Connection returned to pool with 1 advisory locks
"INSIDE TRANSACTION AFTER AJ.retry_job"
Connection returned to pool with 1 advisory locks
"AFTER TRANSCATION"

[bensheldon]

This uses joinable: false which seems relevant.

I can admit I don't fully understand nested transactions in Rails; I probably copied that from writing applications.

Short term: I wonder if just removing the joinable/requires_new would fix this (I see many other instances in GoodJob where I didn't use those options

Long term: I noticed this is a place where the connection-based advisory lock should be replaced with a transaction-based one.

[bensheldon]

I can remove that right now and see what happens 🤞🏻