fix: avoid race condition when building chain processors

[byte-bandit]

Background

I'm surprised this hasn't flared up before. Pigeons will usually "build processors" before every operation, meaning they will ask Paloma for the latest chain configuration and compare their local setup. In case there's a difference, they will rebuild their local setup to match what's asked for by Paloma.

The code for this was not written with concurrency in mind, but is part of most of our background tasks which all run concurrently. Therefore, updating the internal state would interfere with each other, leading to contradicting results, a ton of update calls, invalid and incomplete data in the valsets and more.

This change adds a read/write mutual exclusion lock around the code in question, ensuring that reads stay performant, but don't interfere with updating the internal state any longer.

Testing completed

• test coverage exists or has been added/updated
• tested in a private testnet

Breaking changes

• I have checked my code for breaking changes

[taariq]

To reflect: we now have a new set of mutex locks to prevent race conditions on getting chain infos. Yes, surprised this was not an issue before. Any tests needed here?

[byte-bandit]

To reflect: we now have a new set of mutex locks to prevent race conditions on getting chain infos. Yes, surprised this was not an issue before. Any tests needed here?

@taariq There's tests in place that verify the func works as expected.