🐛 Fakeclient: Fix TOCTOU races #2980
Conversation
The fake client currently has a number of time of check time of use races, where it fetches an object to determine what to do in a mutating operation. The problem is that the object might change in between fetching it and doing the mutating operation. Most notably, this happens when: * Patching is done in parallel. Only one of the patches will succeed, the other ones will fail with a conflict * Updates of objects that allow unconditional updates: All updates will succeed, but not all of them will increment the resource version (i.E dirty writes for the RV) * An update for an object that allows createOnUpdate races with a create or delete * A DeleteAllOf call races with Delete calls * A scale update races with a normal update This change: * Adds tests for all of these cases * Fixes them by adding a lock around the write operations, including their read part, if any
k8s-ci-robot
added
cncf-cla: yes
|
/cc @vincepri @sbueringer |
|
Thank you very much!! /lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 5ff547602e5e17cac4e046740f836e186406b5c2
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alvaroaleman, sbueringer The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/cherrypick release-0.19 |
|
@alvaroaleman: new pull request created: #2992 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
The fake client currently has a number of time of check time of use races, where it fetches an object to determine what to do in a mutating operation. The problem is that the object might change in between fetching it and doing the mutating operation. Most notably, this happens when:
This change:
Note: This patch is a more complete alternate to #2979. Initially I thought that this change might be too breaking to backport, because tests tend to converge extremely closely to any behavior of the fakeclient regardless of it being correct or not, but I think it should be fine. The fact that it took years for someone to notice a very basic issue where concurrent patching doesn't work correctly is also indicative of the fact that the fake client seems to be rarely used concurrently and only then will this change result in any change of behavior.
Fixes #2977
Fixes #2960
/cc @akutz