Resolve race condition between CARM ConfigMap and reconciler for annotated namespaces #138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a-hilaly
force-pushed
the
carm-race-condition/fix
branch
2 times, most recently
from
63afd91
to
eb73742
Compare
…tated namespaces In certain scenarios, where a user deploys a resource to a namespace annotated with a specific ownner accountID, a race condition was identified between the reconciler and the CARM (Cross Account Resource Management) `ConfigMap`. This race condition resulted in the controller setting an empty roleARN, preventing the aws-sdk-go client from pivoting (calling `STS::AssumeRole`) and managing resourecs in the correct account. Instead, resources were inadvertently managed in the default account instead of the namespace assigned account. This issue stemmed from the initial implementation of the CARM feature, where the method responsible for retrieving the accountID from the cache, didn't not properly verify the existance and content of the CARM configMap and instead returned an empty stringy when these conditions were not satisfied. This led to selection of the default account (empty `RoleARN` for resource management. Although these scenarios are rare, they can occur in clusters with a significantly high number of namespaces, causing a delay between naemsapce/configmap events and the informer's event handlers. This patch addresses the race issue by implementing two main things: - Proper error propagation: an error is no propagated when a `ConfigMap` is missing or when an accountID entry is missing in the `ConfigMap`. This helps the reconciler make the right decision on how to handle these cases. - Improved error handling: The reconciler now carefully handles these errors and requeues whenever a user has issued an owneraccountid-annotated namespace but the Configmap is not create or properly propagated. Signed-off-by: Amine Hilaly <hilalyamine@gmail.com>
a-hilaly
force-pushed
the
carm-race-condition/fix
branch
from
eb73742
to
91ccfed
Compare
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: a-hilaly, yuxiang-zhang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ndbhat
pushed a commit
to ndbhat/ack-runtime
that referenced
this pull request
Apr 16, 2024
…tated namespaces (aws-controllers-k8s#138) Addresses aws-controllers-k8s/community#2011 In certain scenarios, where a user deploys a resource to a namespace annotated with a specific ownner accountID, a race condition was identified between the reconciler and the CARM (Cross Account Resource Management) `ConfigMap`. This race condition resulted in the controller setting an empty roleARN, preventing the aws-sdk-go client from pivoting (calling `STS::AssumeRole`) and managing resourecs in the correct account. Instead, resources were inadvertently managed in the default account instead of the namespace assigned account. This issue stemmed from the initial implementation of the CARM feature, where the method responsible for retrieving the accountID from the cache, didn't not properly verify the existance and content of the CARM configMap and instead returned an empty stringy when these conditions were not satisfied. This led to selection of the default account (when an empty `RoleARN` is returned )for resource management. Although these scenarios are rare, they can occur in clusters with a significantly high number of namespaces, causing a delay between naemsapce/configmap events and the informer's event handlers. This patch addresses the race issue by implementing two main things: - Proper error propagation: an error is no propagated when a `ConfigMap` is missing or when an accountID entry is missing in the `ConfigMap`. This helps the reconciler make the right decision on how to handle these cases. - Improved error handling: The reconciler now carefully handles these errors and requeues whenever a user has issued an owneraccountid-annotated namespace but the Configmap is not create or properly propagated. Signed-off-by: Amine Hilaly <hilalyamine@gmail.com> By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses aws-controllers-k8s/community#2011
In certain scenarios, where a user deploys a resource to a namespace
annotated with a specific ownner accountID, a race condition was
identified between the reconciler and the CARM (Cross Account Resource
Management)
ConfigMap. This race condition resulted in the controllersetting an empty roleARN, preventing the aws-sdk-go client from pivoting
(calling
STS::AssumeRole) and managing resourecs in the correctaccount. Instead, resources were inadvertently managed in the default
account instead of the namespace assigned account.
This issue stemmed from the initial implementation of the CARM feature,
where the method responsible for retrieving the accountID from the
cache, didn't not properly verify the existance and content of the CARM
configMap and instead returned an empty stringy when these conditions
were not satisfied. This led to selection of the default account (when an
empty
RoleARNis returned )for resource management.Although these scenarios are rare, they can occur in clusters with a
significantly high number of namespaces, causing a delay between
naemsapce/configmap events and the informer's event handlers.
This patch addresses the race issue by implementing two main things:
ConfigMapis missing or when an accountID entry is missing in the
ConfigMap.This helps the reconciler make the right decision on how to handle
these cases.
errors and requeues whenever a user has issued an
owneraccountid-annotated namespace but the Configmap is not create or
properly propagated.
Signed-off-by: Amine Hilaly hilalyamine@gmail.com
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.