Add safe guard to reduce risk when a large number of nodes are tainted

[johscheuer]

Description

Add safe guard to reduce risk when a large number of nodes are tainted. The idea is to prevent the operator from automatically replacing any process groups on tainted nodes if too many nodes in the cluster are tainted. The MaxConcurrentReplacements is still in place, so this is basically a second safe guard.

Type of change

Please select one of the options below.

• New feature (non-breaking change which adds functionality)

Discussion

Testing

Updated the unit testing setup for the taint feature and added additional tests.

Documentation

Added in the cluster docs.

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: fafdf24
• Duration 2:20:54
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

Did not dig at test refactor, but the changes look good to me other than the comments!

[johscheuer]

Merging as I only changed comments in the second commit.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 11df00f
• Duration 2:31:03
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)