Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rails 6.0.3: *** stack smashing detected *** #316

Open
jfahrenkrug opened this issue Feb 10, 2022 · 3 comments
Open

Rails 6.0.3: *** stack smashing detected *** #316

jfahrenkrug opened this issue Feb 10, 2022 · 3 comments

Comments

@jfahrenkrug
Copy link

Hi,

we have an old-ish Rails application that we are trying to upgrade to Rails 6. We have everything working, but we have a couple of spots where we use Parallel.map to iterate over chunks of DB ids so we can execute sub-queries in parallel (it's a complex data model and this was done to speed up the initial generation of a large JSON response that is subsequently cached). This worked fine in Rails 4 with Unicorn, but in Rails 6 we get this error (tried it both with Puma and with Unicorn):

#<Thread:0x000055ffc2124090 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffc21243b0 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffbd8db9a0 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
#<Thread:0x000055ffc2124220 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep'/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': : stack level too deepstack level too deep ( (SystemStackErrorSystemStackError))

	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffc2124540 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
#<Thread:0x000055ffc21246d0 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
#<Thread:0x000055ffbd8dab90 /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep> terminated with exception (report_on_exception is true):
	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
/usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep': stack level too deep (SystemStackError)
	from /usr/local/bundle/gems/activerecord-6.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `block in spawn_thread'
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
I, [2022-02-10T18:49:43.804366 #14]  INFO -- : [461a9905-5681-47f9-b18a-59048d3e1310] Completed 500 Internal Server Error in 63460ms (ActiveRecord: 2768.6ms | Allocations: 106815)
F, [2022-02-10T18:49:43.805046 #14] FATAL -- : [461a9905-5681-47f9-b18a-59048d3e1310]
[461a9905-5681-47f9-b18a-59048d3e1310] Parallel::DeadWorker (Parallel::DeadWorker):
[461a9905-5681-47f9-b18a-59048d3e1310]
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:78:in `rescue in work'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:75:in `work'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:405:in `block (4 levels) in work_in_processes'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:518:in `with_instrumentation'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:404:in `block (3 levels) in work_in_processes'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:392:in `loop'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:392:in `block (2 levels) in work_in_processes'
[461a9905-5681-47f9-b18a-59048d3e1310] parallel (1.20.1) lib/parallel.rb:215:in `block (4 levels) in in_threads'
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
*** stack smashing detected ***: terminated
I, [2022-02-10T18:49:46.981924 #15]  INFO -- : [ebab2068-1248-466e-980c-f2a08941747a] Completed 500 Internal Server Error in 64306ms (ActiveRecord: 2768.4ms | Allocations: 286933)
F, [2022-02-10T18:49:46.982835 #15] FATAL -- : [ebab2068-1248-466e-980c-f2a08941747a]
[ebab2068-1248-466e-980c-f2a08941747a] Parallel::DeadWorker (Parallel::DeadWorker):
[ebab2068-1248-466e-980c-f2a08941747a]
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:78:in `rescue in work'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:75:in `work'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:405:in `block (4 levels) in work_in_processes'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:518:in `with_instrumentation'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:404:in `block (3 levels) in work_in_processes'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:392:in `loop'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:392:in `block (2 levels) in work_in_processes'
[ebab2068-1248-466e-980c-f2a08941747a] parallel (1.20.1) lib/parallel.rb:215:in `block (4 levels) in in_threads'

The code in our Rails app that is causing this looks like this (slightly modified for clarity):

ids = ArticleGroupConnection.where(article_group_id: ag_id).pluck(:trip_id)
@all_trips = TripLight.includes(:trip_versions, :trip_dates).order('number')
      
count_all = ids.size
count_done = 0

group_size = 70

ids.each_slice(group_size) do |group| 
  count_done += group_size 
  trip_share = @all_trips.where(id: group)
  
   trips_data << Parallel.map(trip_share, in_processes: processcount) do |a_trip|

      Rails.application.executor.wrap do # tried it with and without this, didn't make a difference
        begin
          hash = a_trip.full_hash_v1(false, brand, true) # this is an expensive method with several AR subqueries
          hash
        rescue => err
          Rails.logger.error "######################## ERROR ########################"
          Rails.logger.error  ['api_data', 'each', err.inspect].join(', ')
          Rails.logger.error err.try(:full_message)
          Rails.logger.error err.try(:message)
          Rails.logger.error (err.try(:backtrace) || []).join("\n")
          Rails.logger.error "######################## ERROR END ########################"
          nil
      end
    end
  end
end

When I replace trips_data << Parallel.map(trip_share, in_processes: processcount) do |a_trip| with trips_data << trip_share.map do |a_trip|, everything's fine.

Any idea why this could be happening or how to solve this? Just for reference (and unrelated to parallel), someone else ran into a similar error (but without the stack smashing) after upgrading from Rails 4 to 6: rails/rails#40698

Thanks!
@grosser
Copy link
Owner

grosser commented Feb 10, 2022 •
edited
Loading

try these https://github.com/grosser/parallel#activerecord
if that does not work, then no idea 😞

@jfahrenkrug
Copy link
Author

@grosser Thank you for the reply. Maybe it has something to do with the DB adapter we are using in connection with Rails 6 (https://github.com/ibmdb/ruby-ibmdb). What worries me is that it's not simply an error about a missing DB connection, but a low-level memory/stack overflow crash ("stack smashing").

@grosser
Copy link
Owner

grosser commented Jul 25, 2023

got no idea how this happens, would recommend trying plain threads/forks ... I'd assume it also happens there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@grosser @jfahrenkrug