Workers Seem to wait until all workers complete before grabbing more work?

[jonmchan]

I am using parallel :in_processes and the work is divided very unevenly with certain units of work taking 3-5 minutes to process while others takes only seconds. I am not sure if it is an error on my part or if parallel works by waiting the group of all workers to be done before grabbing new data. I do have mysql DB issues thrown in too, so it is hard for me to tell, but the behavior I'm seeing is:

4 processes:

Job 1 (large job), job 2, job 3, job 4 start

Job 1 hangs arounds for a long time working and it doesn't seem that job 5, 6, 7, and 8 start until job 1 finishes. Is there any way for me to always have 4 processes running simultaneously?

Is this how parallel works or am I running into issues with DB connections like I've seen in other issues here?

[grosser]

You need to deal with db connections, for example disconnect and then reconnect inside.
Jobs don't wait for groups to finish, all workers are independent.

Parallel.map([1,2,3,4,5], in_processes: 2){ |i| puts i; sleep 5-i; puts i }
2
1
2
3
1
4
4
3
5
5

[jonmchan]

Would it be ok if I just create a new DB connection and leave the parent connection there and just not use it? Is it absolutely imperative to disconnect the DB connection? I'm using sequel DB pool.

Could you explain a little more about the DB issue? How come some of my DB connections get through while others don't when I don't disconnect and reconnect inside?