Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel compatible with SQLite? #127

Open
duffyjp opened this issue Apr 8, 2015 · 3 comments
Open

Parallel compatible with SQLite? #127

duffyjp opened this issue Apr 8, 2015 · 3 comments

Comments

@duffyjp
Copy link
Contributor

duffyjp commented Apr 8, 2015

I use Parallel extensively, but ran into an issue in a new project where any code within the Parallel block has no access what-so-ever to ActiveRecord when using SQLite.

I tried the three ActiveRecord workarounds on the README, and they didn't help. It does work in MySQL (dev env), but I use SQLite for testing.

Here's my method with Parallel:

def import
  raise "This file needs to be parsed" unless parsed?
  raise "This file has already been imported" if imported?

  # If any exception is raised, everything will be rolled back.
  ActiveRecord::Base.transaction do
    workbook = ::RubyXL::Parser.parse(file.current_path)
    Parallel.each(workbook.worksheets, in_processes: 3) do |sheet|
      import_sheet(sheet)
    end
  end # transaction
  # Return true or false depending on error count
  errors.blank?
end

Within the import_sheet method, all ActiveRecord queries return no records. It's really strange. Even doing in_processes: 1 doesn't work.

I'm working in Rails 4.1.5 and Ruby 2.1.5.

@grosser
Copy link
Owner

grosser commented Apr 12, 2015

only thing that worked for me in mysql land was disconnect + parallel { connect ; do-work }, if that does not work for sqlite then it might just not work ... does opening 2 irb sessions with the same sqlite config work as expected ?

@faraazahmad
Copy link

Happens with me too. I'm using Parallel to run RSpec tests in openSUSE/osem but get the following errors

An error occurred in a `before(:suite)` hook.
Failure/Error: DatabaseCleaner.clean_with(:truncation)

ActiveRecord::StatementInvalid:
  SQLite3::BusyException: database is locked: DELETE FROM "roles";
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/statement.rb:108:in `step'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/statement.rb:108:in `block in each'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/statement.rb:107:in `loop'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/statement.rb:107:in `each'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/database.rb:149:in `map'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/database.rb:149:in `block in execute'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/database.rb:95:in `prepare'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/database.rb:134:in `execute'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/sqlite3_adapter.rb:317:in `block in execute'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/activesupport-4.2.9/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/sqlite3_adapter.rb:317:in `execute'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/active_record/truncation.rb:94:in `delete_table'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/active_record/truncation.rb:102:in `block in truncate_tables'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/active_record/truncation.rb:102:in `each'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/active_record/truncation.rb:102:in `truncate_tables'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/active_record/truncation.rb:208:in `block in clean'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/activerecord-4.2.9/lib/active_record/connection_adapters/abstract_adapter.rb:288:in `disable_referential_integrity'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/active_record/truncation.rb:204:in `clean'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/base.rb:40:in `clean_with'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/configuration.rb:91:in `block in clean_with'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/configuration.rb:91:in `each'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/database_cleaner-1.3.0/lib/database_cleaner/configuration.rb:91:in `clean_with'
# ./spec/support/factory_girl.rb:8:in `block (2 levels) in <top (required)>'
# lib/ruby_tests_main.rb:7:in `test'
# lib/ruby_tests_main.rb:13:in `block in <main>'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:484:in `call_with_index'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:455:in `process_incoming_jobs'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:437:in `block in worker'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:428:in `fork'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:428:in `worker'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:419:in `block in create_workers'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:418:in `each'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:418:in `each_with_index'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:418:in `create_workers'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:358:in `work_in_processes'
# /home/faraaz/.rvm/gems/ruby-2.4.0/gems/parallel-1.11.2/lib/parallel.rb:264:in `map'
# lib/ruby_tests_main.rb:12:in `<main>'
# ------------------
# --- Caused by: ---
# SQLite3::BusyException:
#   database is locked
#   /home/faraaz/.rvm/gems/ruby-2.4.0/gems/sqlite3-1.3.9/lib/sqlite3/statement.rb:108:in `step'

It seems as if SQLite is unable to handle multiple sessions. Is it an issue with Parallel or SQLite?

@grosser
Copy link
Owner

grosser commented Sep 27, 2017

sqllite I'd say ... can try to lock the queries or something like that ... might just not be possible to do parallel queries ... can try with a simple script that you run twice and see if you can reproduce

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants