-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Urgent: InvalidIdentityTokenException with AWS SDK in Version 3.503.0 causes immediate failure (govcloud) #5749
Comments
Hi @jonathon-mcnabb , Thanks for reaching out. Thanks, |
Hey @RanVaknin, Thanks for the quick response. It looks like with the new version this issue is still occurring. For more context, we are running an EKS cluster with an OIDC provider. |
@jonathon-mcnabb could you provide an example of how to initialize the SDK client, including how you get credentials? If you are relying on the default credential chain, please confirm that. |
Hey @kuhe, here's an example of a client init: import { S3Client } from '@aws-sdk/client-s3';
// Create an Amazon S3 service client object.
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment, @typescript-eslint/no-unsafe-call
const s3Client: S3Client = new S3Client({ region: process.env.AWS_REGION });
export { s3Client }; Out stack utilizes EKS with node microservices, using an ODIC provider. The following shows what env variables are defined on the node: AWS_STS_REGIONAL_ENDPOINTS: regional I have confirmed that a token is present at the path. I am not sure if I am relying on the default credential change, though this may be occurring under the hood. |
With this type of initialization, you are using the default Node.js credential provider chain. It looks in a few different places and then it finds and uses the |
Do you have the IAM identity provider in your account with the same URL as the The only way I seem to get |
Not to muddy up the conversation happening here, but I am facing a similar issue with the following credential configuration # .aws/config
[profile target]
region = us-gov-east-1
# .aws/credentials
[default]
source_profile = target
role_arn = [MASKED]
external_id = [MASKED]
region = us-gov-east-1
[target]
aws_access_key_id = [MASKED]
aws_secret_access_key = [MASKED] Python works, CLI works, Node SDK 3.501.0 works. Node SDK 3.503.0 gives me |
@gilesvessey could you try 3.503.1? |
@kuhe, looks like they are in alignment. Here's the token data and the corresponding error message. "aud": [
"sts.amazonaws.com"
],
"exp": 1706813606,
"iat": 1706727206,
"iss": "https://oidc.eks.us-gov-west-1.amazonaws.com/id/[REDACTED_32_CHAR_CODE] "kubernetes.io": {
"namespace": "REDACTED",
"pod": {
"name": "REDACTED ",
"uid": "REDACTED "
},
"serviceaccount": {
"name": " REDACTED ",
"uid": "REDACTED "
}
},
"nbf": REDACTED,
"sub": "REDACTED "
} "message":"No OpenIDConnect provider found in your account for https://oidc.eks.us-gov-west-1.amazonaws.com/id/[REDACTED_32_CHAR_CODE] It also looks like both my org and gilesvessey are both in gov-cloud regions. Perhaps this breaking change only affects gov-cloud? |
I have a fair number of AWS imports in my package file, all of which found a 3.503.1 version EXCEPT sts. So, I made a test script that only exercises the s3 client. Test code const S3Client = require('@aws-sdk/client-s3').S3
const s3 = new S3Client({})
function test() {
return s3.listBuckets({})
.then(res => {
console.log(res)
return res
})
}
test() For clarity, here are all AWS related lines in my package file. In theory my test code should only exercise "@aws-sdk/client-s3": "3.503.1",
"@aws-sdk/lib-dynamodb": "3.503.1",
"@aws-sdk/client-dynamodb": "3.503.1",
"@aws-sdk/util-dynamodb": "3.503.1",
"@aws-sdk/client-sts": "^3.0.0",
"@aws-sdk/client-ec2": "3.503.1",
"@aws-sdk/client-auto-scaling": "3.503.1",
"@aws-sdk/client-ecr": "3.503.1",
"@aws-sdk/client-ecr-public": "3.503.1",
"@aws-sdk/client-lambda": "3.503.1",
"@aws-sdk/client-codecommit": "3.503.1",
"@aws-sdk/client-cloudwatch-events": "3.503.1",
"@aws-sdk/client-cloudwatch-logs": "3.503.1",
"@aws-sdk/credential-providers": "3.503.1",
"@aws-sdk/client-rds": "3.503.1",
"@aws-sdk/client-iam": "3.503.1",
"@aws-sdk/client-codebuild": "3.503.1",
"@aws-sdk/client-efs": "3.503.1",
"@aws-sdk/client-elastic-load-balancing": "3.503.1",
"@aws-sdk/client-elastic-load-balancing-v2": "3.503.1",
"@aws-sdk/client-organizations": "3.503.1",
"@aws-sdk/client-pricing": "3.503.1",
"@aws-sdk/client-opensearch": "3.503.1",
"@aws-sdk/client-ecs": "3.503.1",
"@aws-sdk/client-sqs": "3.503.1",
"@aws-sdk/client-ssm": "3.503.1",
"@aws-sdk/client-cloudwatch": "3.503.1",
"@aws-sdk/client-inspector2": "3.503.1", Result
Now with this setup "@aws-sdk/client-s3": "3.501.0",
"@aws-sdk/lib-dynamodb": "3.501.0",
"@aws-sdk/client-dynamodb": "3.501.0",
"@aws-sdk/util-dynamodb": "3.501.0",
"@aws-sdk/client-sts": "3.501.0",
"@aws-sdk/client-ec2": "3.501.0",
"@aws-sdk/client-auto-scaling": "3.501.0",
"@aws-sdk/client-ecr": "3.501.0",
"@aws-sdk/client-ecr-public": "3.501.0",
"@aws-sdk/client-lambda": "3.501.0",
"@aws-sdk/client-codecommit": "3.501.0",
"@aws-sdk/client-cloudwatch-events": "3.501.0",
"@aws-sdk/client-cloudwatch-logs": "3.501.0",
"@aws-sdk/credential-providers": "3.501.0",
"@aws-sdk/client-rds": "3.501.0",
"@aws-sdk/client-iam": "3.501.0",
"@aws-sdk/client-codebuild": "3.501.0",
"@aws-sdk/client-efs": "3.501.0",
"@aws-sdk/client-elastic-load-balancing": "3.501.0",
"@aws-sdk/client-elastic-load-balancing-v2": "3.501.0",
"@aws-sdk/client-organizations": "3.501.0",
"@aws-sdk/client-pricing": "3.501.0",
"@aws-sdk/client-opensearch": "3.501.0",
"@aws-sdk/client-ecs": "3.501.0",
"@aws-sdk/client-sqs": "3.501.0",
"@aws-sdk/client-ssm": "3.501.0",
"@aws-sdk/client-cloudwatch": "3.501.0",
"@aws-sdk/client-inspector2": "3.501.0", Result
|
@gilesvessey I'm unable to reproduce this. Is the below similar to your setup? Trying to minimize it to the smallest example. I can't test in my setup:
import { STS } from "@aws-sdk/client-sts";
process.env.AWS_PROFILE = "targets-default";
process.env.AWS_REGION = "us-west-2";
const client = new STS({});
console.log(await client.getCallerIdentity({})); // httpStatusCode 200 OK, assumed-role/AdminTestRole |
@jonathon-mcnabb Yes, the token's issuer will correspond to the error message. Rather, and I'm sorry if this is too obvious a question, but do you have this provider in your AWS account? It would be under IAM > Identity Providers and the name of the provider should be the same as the issuer from the token without the |
Ah ok. Thanks for clarifying. Yup, this provider does appear to be in our AWS account. |
Did an EKS redeployment happen that would subtly change the ID value of the provider, that could cause a mismatch? |
@kuhe Hmm, not that I am aware of. I was able to test the below code in us-gov-west-1 and did not have any issues. Same result with 3.504.0. It looks like there isn't a version 3.503.* for @aws-sdk/client-sts? import { STS } from "@aws-sdk/client-sts";
process.env.AWS_PROFILE = "test-profile";
process.env.AWS_REGION = "us-gov-west-1";
const client = new STS({});
console.log(await client.getCallerIdentity({})); // httpStatusCode 200 OK, assumed-role/AWSReservedSSO_AdministratorAccess_....... |
Attempting a rebuild of our service code using version 3.504.0 of the SDK to see if that resolves anything. Will post results here soon. |
@kuhe, it appears the same issue is still occurring on version 3.504.0 |
My credential configuration is different than that in a few ways,
I am going to test the same configuration in us-east-1 to see what happens. Will report back |
@gilesvessey let's move our discussion to #5755, leave this one for updates to the original problem. |
hey @kuhe, Is there any other information I could provide that may be helpful? I would also be happy to jump on a call to dive into things a bit deeper. let me know! |
I would be ok with getting on a call, but I can't post any contact information in this public forum. Could you send an invite to some form of communication? If you don't want to post it here, contact the AWS SDK for JavaScript team via your AWS account manager. |
@kuhe Sounds good. I reached out to our account manager to set something up. In the meantime, I ran a debugger on our source code and I found a clue: In the "/usr/src/app/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js" file, there is an "endpointParams" variable that is returned as part of the "resolveParams" function. When inspecting this value, it had a region set to "us-east-1". When I manually changed this value to my region "us-gov-west-1", we no longer got the "InvalidIdentityTokenException" error and the connection to AWS was successful. I have confirmed that the AWS_REGION env variable is set to us-gov-west-1. {
UseGlobalEndpoint: false,
UseFIPS: false,
Endpoint: undefined,
Region: "us-east-1",
UseDualStack: false,
} Whats interesting is that this only occurs when processing the "AssumeRoleWithWebIdenityCommand". It appears that the region is picked up by the GetObjectCommand, where this is the value of endpointParams:
|
I believe this is the source code where that env variable (endpointParams) can be found: |
Ok, thanks for the clue on where to look. I'm not sure how this managed to work previously, but I think a config change could be needed going forward. With credential providers that involve an AWS service like STS, there's the concept of an outer and inner client. If you use DynamoDB for example, that would be the outer client, and the credentials are supplied from an inner STS client. There are several potential places to set the region. // This sets it on a global level for all AWS SDKs
// However, this does not set the region for any inner STS credential clients.
// As far as I can tell, this has always been the case.
process.env.AWS_REGION = 'us-gov-west-1'; import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
// Initializing with no args uses the default credential chain,
// which may involve one of several inner clients (STS, SSO, etc.)
new DynamoDBClient({}); import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';
// this is the explicit version of the default init, same as above.
new DynamoDBClient({
credentials: fromNodeProviderChain({})
});
// you can also use a specific credential provider instead of the chain.
import { fromTokenFile } from '@aws-sdk/credential-providers';
new DynamoDBClient({
// no inputs needed if environment variables are set for token usage
credentials: fromTokenFile({})
}); Both the outer and inner client's regions can be configured: import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';
new DynamoDBClient({
region: 'us-gov-west-1', // outer client region
credentials: fromNodeProviderChain({
clientConfig: {
region: 'us-gov-west-1', // inner client region
}
})
}); Please try setting the inner client config region as above, giving it either the desired region or setting it to |
Part of the reason for this input redundancy is that our credential providers For SDK initialization that isn't in the default commercial regions, I think you end up needing to configure the region in two places. |
hmm, ok. It will be challenging for us to update in all of the locations this may need to occur. Wouldn't it be expected that the AWS_REGION env variable is sourced for both the outer client AND the inner client? |
There is pre-existing logic in place to default the inner STS clients to us-east-1 instead of the environment region, I believe because that serves as the global commercial region. I will investigate whether I can change this to more logically handle other partitions. |
This should be fixed as of this release: https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.511.0 When initializing a client with no explicit credentials (and therefore using the defaultCredentialProvider) like this: new Client({
region: "govcloud-region"
}); If there is an STS assume role command involved, this inner STS client's region will default to the outer client's resolved region if this region is not in the AWS default partition. |
This issue seems to be strangely closed to #5799 |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread. |
Checkboxes for prior research
Describe the bug
Urgent: Services cannot connect to AWS
InvalidIdentityTokenException with AWS SDK in Version 3.503.0
Environment:
Description:
After upgrading to @aws-sdk/client-s3 and @aws-sdk/client-dynamodb version 3.503.0, we are encountering an InvalidIdentityTokenException when attempting to access S3 and DynamoDB services. The same issue is not observed in previous versions of the package. No changes were made to the source code other than updating the package version.
Reproduction Steps:
Expected Behavior:
Operations on S3 and DynamoDB should succeed without identity token issues, as they did in previous versions of the SDK.
Actual Behavior:
Received an error with the following details:
SDK version number
@aws-sdk/package-name@version, ...
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
v16.20.2
Reproduction Steps
"dependencies": {
"@aws-sdk/client-s3": "^3.501.0",
"@aws-sdk/client-dynamodb": "^3.501.0",
}
Observed Behavior
When connecting to AWS services, the following error is seen.
Received an error with the following details:
Expected Behavior
I expect to be able to utilize AWS SDKs to connect to S3 and DynamoDB.
Possible Solution
No response
Additional Information/Context
No response
The text was updated successfully, but these errors were encountered: