feat: algorithm-queue scale

[nassiharel]

This change is 

[yehiyam]

Reviewed 11 of 11 files at r1.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @nassiharel)

---

core/algorithm-queue/lib/queues-manager.js, line 77 at r1 (raw file):

    async _checkIdleAlgorithms() {
        const queues = Array.from(this._queues.values());
        const isIdle = queues.filter(q => q.isIdle());

You first check this._isIdle and only after you call checkIdle?

---

core/algorithm-queue/lib/queues-manager.js, line 79 at r1 (raw file):

        const isIdle = queues.filter(q => q.isIdle());
        const isStaled = queues.filter(q => q.isStaled());
        await Promise.all(queues.map(a => a.checkIdle()));

why do you call checkIdle here?

---

core/algorithm-queue/lib/queues-manager.js, line 81 at r1 (raw file):

        await Promise.all(queues.map(a => a.checkIdle()));
        await Promise.all(isIdle.map(q => q.pause()));
        await Promise.all(isStaled.map(async q => {

you can call Promise.allSettled and map the results with the error

---

core/algorithm-queue/lib/jobs/consumer.js, line 42 at r1 (raw file):

    }

    async pause() {

no resume?

[nassiharel]

Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @nassiharel and @yehiyam)

---

core/algorithm-queue/lib/queues-manager.js, line 77 at r1 (raw file):

Previously, yehiyam wrote…

You first check this._isIdle and only after you call checkIdle?

If the checkIdle set to true, the _isIdle will be relevant in the next tick (5 sec).

---

core/algorithm-queue/lib/queues-manager.js, line 81 at r1 (raw file):

Previously, yehiyam wrote…

you can call Promise.allSettled and map the results with the error

This is not much cleaner in this case

const all = await Promise.allSettled(isStaled.map(async q => this._removeAction(q.algorithmName, 'idle', false)));
const rejected = all.filter(a => a.status === 'rejected');
if (rejected.length) {
      rejected.forEach(r => {
          log.throttle.error(`error removing idle queue ${q.algorithmName} ${e.message}`, { component }, e);
          });
}

---

core/algorithm-queue/lib/jobs/consumer.js, line 42 at r1 (raw file):

Previously, yehiyam wrote…

no resume?

Once it paused it means it was idle & empty for a long time, so in this case, I want to shut it down anyway.
If there is a demand for that algorithm it will be up again.

[yehiyam]

Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @nassiharel)

[nassiharel]

/deploy