You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are running into an intermittent issue on our Kubernetes pods using kube2iam to provide IAM credentials to containers where the assumed role tries to assume itself
The first thing our pod does is decrypt SOPS secrets using SOPS
We are getting this error message while decrypting:
Failed to get the data key required to decrypt the SOPS file.
Group 0: FAILED
arn:aws:kms:us-east-1:<account-id>:key/<uuid>: FAILED
- | Error decrypting key: NoCredentialProviders: no valid
| providers in chain. Deprecated.
| For verbose messaging see
| aws.Config.CredentialsChainVerboseErrors
arn:aws:kms:us-east-1:<account-id>:key/<uuid>: FAILED
- | Error creating AWS session: Failed to assume role
| "arn:aws:iam::<account-id>:role/service/<role-name>":
| AccessDenied: User:
| arn:aws:sts::<account-id>:assumed-role/<role-name>/<role-session-name>
| is not authorized to perform: sts:AssumeRole on resource:
| arn:aws:iam::<account-id>:role/service/<role-name>
| status code: 403, request id:
| a401448b-6242-46d1-80d7-7e14396b4ad0
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html
We have enabled kube2iam verbose logs and see these logs related to the pod having the error:
— Is this a race condition between kube2iam and SOPS where SOPS tries to assume a role before kube2iam has fully assumed it?
— Is there a way to set the trust relationship of the role to be able to assume itself?
The text was updated successfully, but these errors were encountered:
Try adding an init container that sleeps for a while and see if that solves the problem. If the pod starts up too quickly it may cause problems with the role assumption, however I believe this particular race was fixed a while ago.
If it is not that then see if you can assume the role on a node without kube2iam as a proxy. If not then the problem lies in the role and trust configurations.
We are running into an intermittent issue on our Kubernetes pods using kube2iam to provide IAM credentials to containers where the assumed role tries to assume itself
The first thing our pod does is decrypt SOPS secrets using SOPS
We are getting this error message while decrypting:
We have enabled kube2iam verbose logs and see these logs related to the pod having the error:
Also we're seeing this a lot in our kube2iam pods:
Is this log line expected?
— Is this a race condition between kube2iam and SOPS where SOPS tries to assume a role before kube2iam has fully assumed it?
— Is there a way to set the trust relationship of the role to be able to assume itself?
The text was updated successfully, but these errors were encountered: