Improve the checkClientCompatibility reconciler

[johscheuer]

What would you like to be added/changed?

During version incompatible upgrades the checkClientCompatibility this reconciler will check if there are any incompatible processes with the new desired version. Currently this is implemented in a sub-optimal way that if the check fails, because at least one process is reported as being incompatible, all the further reconcilers will be skipped. That means this issue must be solved fairly quickly to allow the operator to work properly again.

There are different ways how we could solve this e.g. by doing the actual check of checkClientCompatibility inside the updateStatus reconciler and update the cluster.Status in the FoundationDBCluster resource with the result. Based on that information the updateSidecarVersions could update the sidecars or not. The same check would be needed when the fdb-monitor configuration is created, to make sure that the config is only adjusted when the client compatibility check is fine.

[johscheuer]

e.g. when implementing this change we could add a new condition for the failing client compatibility check as mentioned in: #1000.