Synchronize keys between validator client and external signer

[nflaig]

Problem description

It is possible to delegate signing duties to a remote signer by specifying a --externalSigner.url on the validator client. For the validator client to know which pubkeys are managed by the remote signer it is required to either

• specifically define them via --externalSigner.pubkeys flag
• or to have the vc automatically fetch them at startup by setting --externalSigner.fetch flag

The later approach is much more commonly used as it is the most convenient way to set up the validator client with a remote signer. However, the problem is that the keys will get out of sync if new keys are imported (or deleted) into the remote signer as Lodestar only queries the /api/v1/eth2/publicKeys API on startup.

A common workaround for this is to use external tooling / sidecars to keep the keys in sync by calling POST /eth/v1/remotekeys and DELETE /eth/v1/remotekeys on both the validator client and remote signer, or alternatively poll /api/v1/eth2/publicKeys on the remote singer and only call POST /eth/v1/remotekeys on the validator client.

Either way, it's not ideal that this lifecycle has to be managed by external tooling and it likely the best option to let the validator client take care of keeping the keys in sync as it is already able to call the /api/v1/eth2/publicKeys API of the remote signer while it is not guaranteed that the remote signer will be able to send requests tot the validator client as this is currently not a requirement and the connectivity might be one directional in many setups.

Solution description

Implement a solution similar to Nimbus which already polls the remote signer /api/v1/eth2/publicKeys API in a configurable interval to keep the keys in sync.

We could just enable this polling if a user has already set the --externalSigner.fetch flag but instead of just fetching the keys on startup, we would do it every epoch or potentially just every hour (as Nimbus does) and add an additional flag to configure the interval, e.g. --externalsigner.fetchInterval or similar.

The validator client would have to compare the list of pubkeys received from the remote signer against the (remote) pubkeys imported in the validator store and add any missing keys.

It also has to make sure to remove any extraneous keys no longer present on the remote signer as otherwise it would still try to handle duties for those keys but signature requests would fail because the keys are removed from remote signer. It's not a slashing risk in that case due to double signing but would cause a bunch of errors on the validator client side and creates additional overhead for the connected beacon node.

Additional context

Related discussion on discord

[philknows]

Tagged as v1.19 but understand this needs discussion with @g11tech to ensure safety.

[nflaig]

Tagged as v1.19 but understand this needs discussion with @g11tech to ensure safety.

Just to clarify, this does not change the behavior as this is what the sidecar process used by Diva does already, and Nimbus has implemented for a while.

Also we fetch the pubkeys already on startup if --externalSigner.fetch is set, so the only difference this makes is to do it continuously without requiring a restart.