feat(scripts): use redlock to prevent >1 instance of paypal-processor

[chenba]

Because:

• we could easily end up running two instances of the paypal-processor
  during a deploy

This commit:

• use a redis based distributed lock to ensure only one
  paypal-processor can run per env
• add script options to control the lock name and duration, as well as
  completely bypassing the lock

[chenba]

@bbangert one more time? I actually added redlock to the correct package this time. 😓

[pdehaan]

Q: Will a finally() run after a process.exit()? or is process.exit() immediate?
(although that should be a quick experiment for me to run in isolation separately)

[chenba]

hah yeah, I think you are right that that finally won't get executed. I'll fix. Thanks!

[pdehaan]

Doesn't look like it:

Promise.resolve("OK, computer")
  .then(msg => {
    console.log("success:", msg);
    throw new Error("Ruh roh");
  })
  .catch(err => {
    console.error("error:", err.message);
    process.exit(1);
  })
  .finally(() => console.log("I'm still here..."));

node --version # v16.14.2
node index.mjs
# success: OK, computer
# error: Ruh roh

Note that the .finally() log never seems to fire due to the hard process.exit().

[pdehaan]

Deeper and deeper… This seems to do what I'd expect, but not sure if I know what I'm doing:

async function init(e=false) {
  if (e) {
    throw new Error("OH NO!");
  }
  return 0;
}

init(false)
  .then((result) => {
    console.log("then...", result);
    process.exitCode = result;
  })
  .catch((err) => {
    console.error("catch...", err.message);
    process.exitCode = 1;
  })
  .finally(() => {
    console.log("finally...", process.exitCode);
    process.exit(process.exitCode);
  });

I had to swap the .then() and .catch() order, since .catch() swallows errors then proceeds onto the next .then() chain.

But now if the initial init() or .then() block fails, the code should go into the .catch() block and set a non-zero exit code which falls through to the .finally() block which can release locks and set hard exit with your desired exit code.

---

UPDATE: Per the docs, https://nodejs.org/api/process.html#processexitcode, It doesn't sound like you even need to specify the redundant process.exitCode in the process.exit() command in our .finally() block above.

If code is omitted, exit uses either the 'success' code 0 or the value of process.exitCode if it has been set.

[pdehaan]

Hhmm... Not sure this is needed since you already default it to 1 above on L119. 🤷
Unless we should be defaulting the exit status code to 0 above.

[pdehaan]

Is there a chance that the .release() API can throw an error?
Does this need to be wrapped in a try..catch for extra safety?

I guess the risk is that we make it to the finally() block and the lock release throws an error and our process.exit() never gets called (or would terminate with a 0 status code). Although maybe the thrown error would break elsewhere and it's a non-issue.

[chenba]

Thanks. I moved the .release() call into init so that it's in a try block.

[bbangert]

If processing an invoice throws an exception, how is the lock released?

[chenba]

The lock expires after DURATION since acquisition or the most recent extension.

[bbangert]

The example shows:

lock = await lock.extend(timer.elapsed());

I pulled up the source code for extend, and it creates an entirely new lock object that is returned during extend:
https://github.com/mike-marcacci/node-redlock/blob/main/src/index.ts#L410

[chenba]

ooh good catch. 😬 I'll assign the return value to lock.

[chenba]

Opened #12610 as an alternative approach with a auto-extending lock.

[chenba]

Closing this in favor of #12610. @bbangert @pdehaan thank you for all the feedback.