Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

async.queue pause/resume concurrency problem #512

Closed
wants to merge 2 commits into from
Closed

async.queue pause/resume concurrency problem #512

wants to merge 2 commits into from

Conversation

i3fox
Copy link

@i3fox i3fox commented Apr 16, 2014

In the async.queue I found an issue related to restart after a pause (resume) and I try to fix.
It's not simple to explain, please follow these steps:

  1. for concurrency tasks are in execution
  2. I pause the queue and wait for the execution of tasks finishes
  3. I resume queue
  4. Queue restarts but concurrency does not work: only one task is running.

Example of problem:

var async = require("async");
var q = async.queue(function (task, callback) {
    console.log('hello ' + task.name);
    setTimeout(function() {
        callback();
    }, 2000);
}, 5);

q.pause();
q.push({name: '1'});
q.push({name: '2'});
q.push({name: '3'});
q.push({name: '4'});
q.push({name: '5'});
q.push({name: '6'});
q.push({name: '7'});
q.push({name: '8'});
q.push({name: '9'});
q.push({name: '10'});

setTimeout(function () {
    q.resume();
    }, 2500);

q.process();
_each(q.tasks, function(task) {
async.setImmediate(q.process);
});
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait -- this calls process() for each task left in the queue, even if there's 2500+ of them? Surely there's a better way to fix the concurrency. Mabye only call process() n times where n = concurrency.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are right, if the queue contains many tasks the cycle can be heavy.
Could be so:

for (var i = 0; i < q.tasks.length && i < q.concurrency; i++) {
    async.setImmediate(q.process);
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would work a lot better.

@caolan
Copy link
Owner

caolan commented Apr 28, 2014

Can you please add a failing test for this. I think it's a legitimate issue, I just want the patch to be accompanied by a test case.

@elliotbonneville
Copy link

Looks like this bug is (still?) appearing in NPM's package, v0.9.0. Any updates since then?

@ralgara
Copy link

ralgara commented Mar 10, 2015

Still an issue in 0.9.0. In #508, @caolan suggests against calling the callback synchronously. I suppose this means wrapping the callback in either setImmediate or process.nextTick. I have tried both to no avail. Any intention to merge this pull request?

jbmusso added a commit to jbmusso/gremlin-javascript that referenced this pull request May 19, 2015
aearly added a commit that referenced this pull request May 21, 2015
@aearly
Copy link
Collaborator

aearly commented May 21, 2015

I just added a couple tests for this and it seems to work fine. Am I testing what is causing the issue properly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants