RPC redundancy

[efstajas]

The Drips App, Event Processor and API shouldn't go down if our main RPC provider, Infura, goes down. We should implement some RPC redundancy with a secondary RPC provider on a cheap plan with auto-scaling enabled.

The changes below are described for the app specifically, but similar logic should be put in place for Event Processor and API.

Outline:

• Sign up for a paid account with an alternative provider (e.g. Alchemy), choose a low-tier plan, but enable auto-scaling in case the fallback goes into effect during a time of elevated traffic.
• Create a generic RPC API endpoint that we direct all RPC traffic from the app to, e.g. /api/rpc.
• Any RPC hitting the endpoint should first go to Infura. If Infura returns an error or times out (low timeout threshold), it should retry the same request with the backup RPC.
• Return response of backup RPC or error for good if the backup is also faulty.

Important is that we should catch and forward RPC errors from main to Sentry, even if the backup saved the request, so that we get notified.

Status:

• App (refactor: use the Failover Provider #1218)
• Events processor
• Multiplayer API (Integrate failover provider multiplayer#15)
• API (refactor: use the Failover Provider graphql-api#28)
• Setup failover account

[bordumb]

Cross-posting here just so the idea isn't lost...

I saw the recent X/Twitter post that Drips was affected by Infura going down.

I've heard that some teams are using Grove as a fallback for issues like these.

Details below. Pretty good list of others using it.

https://www.grove.city/