Prefer upgraded processes for coordinators when the operator chooses the coordinators

[johscheuer]

Description

Fixes: #2015

Type of change

Please select one of the options below.

• New feature (non-breaking change which adds functionality)

Discussion

As mentioned in the referenced issue, during an upgrade it can happen that the operator chooses new coordinators for the Pod that is currently "updated" (deleted and created). In the previous setup the operator didn't differentiate between upgraded processes and processes to be upgraded. Choosing preferable upgraded processes as coordinators should reduce the overall recoveries caused by coordinator changes.

In addition we opted to prefer log/transaction processes as coordinators as those are not going though the rolling recreation phase but instead all are replaced.

Testing

Updated unit tests and current e2e test should cover all important cases.

Documentation

I can update the docs in another PR. I have to revisit them anyways.

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 0cd7bce
• Duration 2:14:34
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 0cd7bce
• Duration 2:22:43
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 0cd7bce
• Duration 2:22:43
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

Summarizing 1 Failure:
  [FAIL] Operator when the operator is allowed to reset the maintenance zone [It] should reset the maintenance mode once the Pod was restarted [e2e, pr]
  /codebuild/output/src3153794519/src/github.com/FoundationDB/fdb-kubernetes-operator/e2e/test_operator/operator_test.go:2072

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 0cd7bce
• Duration 2:00:46
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 0cd7bce
• Duration 2:22:08
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 20b7861
• Duration 2:12:54
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

lgtm as long as it's not possible for /var/-prefix command line to be used for a fully upgraded process

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 092860b
• Duration 2:32:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 9631c1b
• Duration 2:25:08
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)