Only release the lock when the cluster is reconciled

[johscheuer]

Description

In the past we added code to release the lock when the operation was performed. Doing that increases the risk of race conditions when multiple operator instances are managing a multi-region (or three data hall) FDB cluster. In order to reduce the risk of those race conditions the operator is releasing the lock only when the cluster is reconciled (or when the lock is timed out).

Type of change

Please select one of the options below.

• Bug fix (non-breaking change which fixes an issue)

Discussion

The general idea of the locking mechanism is to reduce the operations executed in parallel on the FDB cluster. When the lock is directly released after the operation we have the risk that another operator instance is directly doing another operation that could be disruptive, e.g. excluding processes.

Testing

e2e test will be running by CI.

Documentation

Will be updated.

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 5a6038c
• Duration 2:58:37
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 15b4edc
• Duration 2:56:52
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

LGTM, though I was thinking it could be nice to output the cluster.LockDuration in the TakeLock failure output for users to see how long they should expect to wait in the case of a problem (though I don't know how many people use multi operator setups)

[johscheuer]

LGTM, though I was thinking it could be nice to output the cluster.LockDuration in the TakeLock failure output for users to see how long they should expect to wait in the case of a problem (though I don't know how many people use multi operator setups)

That makes sense, I'll add it 👍

[johscheuer]

The logger already contains the information, so there is no need to update the code: https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/fdbclient/lock_client.go#L118-L147

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: da4fb90
• Duration 3:35:16
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: e841c46
• Duration 3:22:57
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 6d18fea
• Duration 3:23:10
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)