Connection keep alive mechanism broken on key changes

[eskimor]

It seems that we keep track of connection requests by AuthorityDiscoveryId in validator discovery and we issue add/remove to peer set requests based on that here. Problem is, if an authority changed its key at the session boundary, then different AuthorityDiscoveryIds would match to the same PeerId, resulting in the connection to be dropped, although it should be maintained.

This happens as the same PeerId/address would be added and removed immediately afterwards.

@ordian is this analysis correct or am I missing something?

[ordian]

Hmm, you're right!

However, there is also another issue with key rotations. The problem is that the authority discovery cache might remove the entry for the old AuthorityId link, which might lead to a leak instead of removal.

[eskimor]

Right! Using the cache for anything else than the initial discovery is just racy and prone to bugs. That cache is not immutable, so we should not treat it as if it were.