Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deadlock issue from threads autoloading and requiring the same dependency #688

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

evanlok
Copy link

@evanlok evanlok commented Dec 7, 2022

Spring occasionally crashes on boot due to deadlock errors that occur from calling require and autoload of Bundler::SharedHelpers in separate threads.

Ruby bug: https://bugs.ruby-lang.org/issues/15598

Error stack trace:

/usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:12:in `<module:Bundler>': No live threads left. Deadlock?
2 threads, 2 sleeps current:0x000055ccc7d6b950 main thread:0x000055ccc7941f00
* #<Thread:0x000055ccc796ed78 sleep_forever>
   rb_thread_t:0x000055ccc7941f00 native:0x00007fe7557b4f00 int:0
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:12:in `<module:Bundler>'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:11:in `<top (required)>'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `require_relative'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `<top (required)>'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:37:in `<module:Spring>'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:4:in `<top (required)>'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:87:in `preload'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:166:in `serve'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:148:in `block in run'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `loop'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `run'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application/boot.rb:19:in `<top (required)>'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   -e:1:in `<main>'
* #<Thread:0x000055ccc7c52620 /usr/local/bundle/gems/spring-4.1.0/lib/spring/failsafe_thread.rb:6 sleep_forever>
   rb_thread_t:0x000055ccc7d6b950 native:0x00007fe751eb5700 int:0
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler.rb:462:in `default_gemfile'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:14:in `gemfile'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:63:in `find_project_root'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:53:in `project_root_path'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:44:in `application_root_path'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/env.rb:20:in `root'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/env.rb:67:in `app_name'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:36:in `app_name'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application/boot.rb:15:in `block in <top (required)>'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/process_title_updater.rb:44:in `value'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/process_title_updater.rb:14:in `block in run'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/failsafe_thread.rb:8:in `block in failsafe_thread'
 (fatal)
	from /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:11:in `<top (required)>'
	from /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `require_relative'
	from /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `<top (required)>'
	from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:37:in `<module:Spring>'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:4:in `<top (required)>'
	from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:87:in `preload'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:166:in `serve'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:148:in `block in run'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `loop'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `run'
	from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application/boot.rb:19:in `<top (required)>'
	from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
	from -e:1:in `<main>'

@rafaelfranca
Copy link
Member

How do we reproduce this deadlock? I could not reproduce it.

@evanlok
Copy link
Author

evanlok commented Mar 20, 2023

It's difficult to reproduce reliably since it's a race condition between threads. You may be able to reproduce it by writing a script that creates 2 threads: 1 to require shared_helpers and 1 to autoload shared_helpers then invoke that script several times in a loop. We saw this issue occur somewhat often in our CI builds (at least once per day) until this fix was applied. Note that we were running on Ruby 2.7.2. The root cause may have been addressed in newer versions of Ruby although the Ruby bug does not indicate any resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants