[DPE-5311] Enable DCS failsafe mode

[dragomirp]

Issue

Patroni depends on the K8s API and resources to maintain the leader lock. Primary will step down during a temporary K8s API outage

Solution

Enable DCS failsafe mode. If the K8s API (DCS) is not available, Patroni will try to maintain the current primary if it can connect to all cluster members

Documentation: https://patroni.readthedocs.io/en/latest/dcs_failsafe_mode.html#dcs-failsafe-mode

Related #669 and #616

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.71%. Comparing base (fde548d) to head (0f7c923).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #677   +/-   ##
=======================================
  Coverage   70.71%   70.71%           
=======================================
  Files          11       11           
  Lines        2950     2950           
  Branches      513      513           
=======================================
  Hits         2086     2086           
  Misses        754      754           
  Partials      110      110           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[taurus-forever]

@7annaba3l we are pending your ACK to merge this.

I believe we should enable failsafe, @delgod had doubts about the split brain but manuals says it is safe:
https://patroni.readthedocs.io/en/latest/dcs_failsafe_mode.html#dcs-failsafe-mode

If the failsafe mode is enabled and the leader lock update in DCS failed due to
reasons different from the version/value/index mismatch,
Postgres may continue to run as a primary if it can access all
known members of the cluster via Patroni REST API.
...
The primary will execute the failsafe code and contact all known replicas.
These replicas will use this information as an indicator that the primary is
alive and will not start the leader race even if the leader lock in DCS has expired.

4 @dragomirp : I want to merge this and fix #669 and #616.

[delgod]

after careful check of implementation details - everything looks safe (even in upgrade case) and I believe we should enable this option.

[taurus-forever]

Mykola approved, Mohamed notified in MM. LGTM from me. Thank you!