Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Urgent: InvalidIdentityTokenException with AWS SDK in Version 3.503.0 causes immediate failure (govcloud) #5749

Closed
3 tasks done
jonathon-mcnabb opened this issue Jan 30, 2024 · 30 comments
Assignees
Labels
bug This issue is a bug. p1 This is a high priority issue

Comments

@jonathon-mcnabb
Copy link

jonathon-mcnabb commented Jan 30, 2024

Checkboxes for prior research

Describe the bug

Urgent: Services cannot connect to AWS

InvalidIdentityTokenException with AWS SDK in Version 3.503.0

Environment:

AWS SDK Version: 3.503.0
Services we use that are affected: Amazon S3, Amazon DynamoDB (likely affects others)

Description:
After upgrading to @aws-sdk/client-s3 and @aws-sdk/client-dynamodb version 3.503.0, we are encountering an InvalidIdentityTokenException when attempting to access S3 and DynamoDB services. The same issue is not observed in previous versions of the package. No changes were made to the source code other than updating the package version.

Reproduction Steps:

Upgrade @aws-sdk/client-s3 to version 3.503.0.
Upgrade @aws-sdk/client-dynamodb to version 3.503.0.
Attempt to perform standard operations on S3 and DynamoDB.
Observe the InvalidIdentityTokenException.

Expected Behavior:
Operations on S3 and DynamoDB should succeed without identity token issues, as they did in previous versions of the SDK.

Actual Behavior:
Received an error with the following details:

Error Name: InvalidIdentityTokenException
Fault: client
HTTP Status Code: 400
Request ID: [REDACTED]
Error Type: Sender
Error Code: InvalidIdentityToken
Error Message: No OpenIDConnect provider found in your account for [REDACTED URL]

SDK version number

@aws-sdk/package-name@version, ...

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v16.20.2

Reproduction Steps

"dependencies": {
"@aws-sdk/client-s3": "^3.501.0",
"@aws-sdk/client-dynamodb": "^3.501.0",
}

Observed Behavior

When connecting to AWS services, the following error is seen.

Received an error with the following details:

Error Name: InvalidIdentityTokenException
Fault: client
HTTP Status Code: 400
Request ID: [REDACTED]
Error Type: Sender
Error Code: InvalidIdentityToken
Error Message: No OpenIDConnect provider found in your account for [REDACTED URL]

Expected Behavior

I expect to be able to utilize AWS SDKs to connect to S3 and DynamoDB.

Possible Solution

No response

Additional Information/Context

No response

@jonathon-mcnabb jonathon-mcnabb added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 30, 2024
@jonathon-mcnabb jonathon-mcnabb changed the title Urgent: VInvalidIdentityTokenException with AWS SDK in Version 3.503.0 causes immediate failure Urgent: InvalidIdentityTokenException with AWS SDK in Version 3.503.0 causes immediate failure Jan 30, 2024
@RanVaknin RanVaknin self-assigned this Jan 30, 2024
@RanVaknin
Copy link
Contributor

Hi @jonathon-mcnabb ,

Thanks for reaching out.
We just released a fix for some other breaking changes. Can you pull 3.503.1 and see if it fixes your issue?

Thanks,
Ran~

@RanVaknin RanVaknin added response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. p1 This is a high priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Jan 30, 2024
@jonathon-mcnabb
Copy link
Author

Hey @RanVaknin,

Thanks for the quick response. It looks like with the new version this issue is still occurring.

For more context, we are running an EKS cluster with an OIDC provider.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Jan 31, 2024
@kuhe kuhe self-assigned this Jan 31, 2024
@kuhe kuhe added the investigating Issue is being investigated and/or work is in progress to resolve the issue. label Jan 31, 2024
@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

@jonathon-mcnabb could you provide an example of how to initialize the SDK client, including how you get credentials?

If you are relying on the default credential chain, please confirm that.

@kuhe kuhe added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Jan 31, 2024
@jonathon-mcnabb
Copy link
Author

jonathon-mcnabb commented Jan 31, 2024

Hey @kuhe, here's an example of a client init:

import { S3Client } from '@aws-sdk/client-s3';
// Create an Amazon S3 service client object.
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment, @typescript-eslint/no-unsafe-call
const s3Client: S3Client = new S3Client({ region: process.env.AWS_REGION });
export { s3Client };

Out stack utilizes EKS with node microservices, using an ODIC provider. The following shows what env variables are defined on the node:

AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_ROLE_ARN: arn:aws-us-gov:iam::[REDACTED_ACCOUNT]:role/[REDACTED_ROLE]
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

I have confirmed that a token is present at the path.

I am not sure if I am relying on the default credential change, though this may be occurring under the hood.

@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

With this type of initialization, you are using the default Node.js credential provider chain. It looks in a few different places and then it finds and uses the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE variables to read a token file and then make a request with STS::assumeRoleWithWebIdentity to get credentials to then make the subsequent S3 requests.

@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

Do you have the IAM identity provider in your account with the same URL as the iss field in your token?

The only way I seem to get No OpenIDConnect provider found in your account for [URL] is if I set [URL] to a provider that doesn't exist in my account.

@gilesvessey
Copy link

gilesvessey commented Jan 31, 2024

Not to muddy up the conversation happening here, but I am facing a similar issue with the following credential configuration

# .aws/config
[profile target]
region = us-gov-east-1
# .aws/credentials
[default]
source_profile = target
role_arn = [MASKED]
external_id = [MASKED]
region = us-gov-east-1
[target]
aws_access_key_id = [MASKED]
aws_secret_access_key = [MASKED]

Python works, CLI works, Node SDK 3.501.0 works. Node SDK 3.503.0 gives me InvalidClientTokenId: The security token included in the request is invalid.

@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

@gilesvessey could you try 3.503.1?

@jonathon-mcnabb
Copy link
Author

jonathon-mcnabb commented Jan 31, 2024

@kuhe, looks like they are in alignment.

Here's the token data and the corresponding error message.

  "aud": [
    "sts.amazonaws.com"
  ],
  "exp": 1706813606,
  "iat": 1706727206,
  "iss": "https://oidc.eks.us-gov-west-1.amazonaws.com/id/[REDACTED_32_CHAR_CODE]  "kubernetes.io": {
    "namespace": "REDACTED",
    "pod": {
      "name": "REDACTED ",
      "uid": "REDACTED "
    },
    "serviceaccount": {
      "name": " REDACTED ",
      "uid": "REDACTED "
    }
  },
  "nbf": REDACTED,
  "sub": "REDACTED "
}

"message":"No OpenIDConnect provider found in your account for https://oidc.eks.us-gov-west-1.amazonaws.com/id/[REDACTED_32_CHAR_CODE]

It also looks like both my org and gilesvessey are both in gov-cloud regions. Perhaps this breaking change only affects gov-cloud?

@gilesvessey
Copy link

gilesvessey commented Jan 31, 2024

I have a fair number of AWS imports in my package file, all of which found a 3.503.1 version EXCEPT sts. So, I made a test script that only exercises the s3 client.

Test code

const S3Client = require('@aws-sdk/client-s3').S3
const s3 = new S3Client({})

function test() {
  return s3.listBuckets({})
    .then(res => {
      console.log(res)
      return res
    })
}

test()

For clarity, here are all AWS related lines in my package file. In theory my test code should only exercise client-s3

    "@aws-sdk/client-s3": "3.503.1",
    "@aws-sdk/lib-dynamodb": "3.503.1",
    "@aws-sdk/client-dynamodb": "3.503.1",
    "@aws-sdk/util-dynamodb": "3.503.1",
    "@aws-sdk/client-sts": "^3.0.0", 
    "@aws-sdk/client-ec2": "3.503.1",
    "@aws-sdk/client-auto-scaling": "3.503.1",
    "@aws-sdk/client-ecr": "3.503.1",
    "@aws-sdk/client-ecr-public": "3.503.1",
    "@aws-sdk/client-lambda": "3.503.1",
    "@aws-sdk/client-codecommit": "3.503.1",
    "@aws-sdk/client-cloudwatch-events": "3.503.1",
    "@aws-sdk/client-cloudwatch-logs": "3.503.1",
    "@aws-sdk/credential-providers": "3.503.1",
    "@aws-sdk/client-rds": "3.503.1",
    "@aws-sdk/client-iam": "3.503.1",
    "@aws-sdk/client-codebuild": "3.503.1",
    "@aws-sdk/client-efs": "3.503.1",
    "@aws-sdk/client-elastic-load-balancing": "3.503.1",
    "@aws-sdk/client-elastic-load-balancing-v2": "3.503.1",
    "@aws-sdk/client-organizations": "3.503.1",
    "@aws-sdk/client-pricing": "3.503.1",
    "@aws-sdk/client-opensearch": "3.503.1",
    "@aws-sdk/client-ecs": "3.503.1",
    "@aws-sdk/client-sqs": "3.503.1",
    "@aws-sdk/client-ssm": "3.503.1",
    "@aws-sdk/client-cloudwatch": "3.503.1",
    "@aws-sdk/client-inspector2": "3.503.1",

Result

node s3-test.js 
/Users/gvessey/src/giles-509/tools/node_modules/@smithy/smithy-client/dist-cjs/index.js:838
  const response = new exceptionCtor({
                   ^

InvalidClientTokenId: The security token included in the request is invalid.
    at throwDefaultError (/Users/gvessey/src/giles-509/tools/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:20)
    at /Users/gvessey/src/giles-509/tools/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:5
    at de_AssumeRoleCommandError (/Users/gvessey/src/giles-509/tools/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:375:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gvessey/src/giles-509/tools/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async Object.roleAssumer (/Users/gvessey/src/giles-509/tools/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:1516:43)
    at async /Users/gvessey/src/giles-509/tools/node_modules/@smithy/property-provider/dist-cjs/index.js:79:27 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '56f68396-f357-48d3-aaf1-f45af4eca2f2',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Type: 'Sender',
  Code: 'InvalidClientTokenId'
}

Node.js v18.18.2

Now with this setup

    "@aws-sdk/client-s3": "3.501.0",
    "@aws-sdk/lib-dynamodb": "3.501.0",
    "@aws-sdk/client-dynamodb": "3.501.0",
    "@aws-sdk/util-dynamodb": "3.501.0",
    "@aws-sdk/client-sts": "3.501.0",
    "@aws-sdk/client-ec2": "3.501.0",
    "@aws-sdk/client-auto-scaling": "3.501.0",
    "@aws-sdk/client-ecr": "3.501.0",
    "@aws-sdk/client-ecr-public": "3.501.0",
    "@aws-sdk/client-lambda": "3.501.0",
    "@aws-sdk/client-codecommit": "3.501.0",
    "@aws-sdk/client-cloudwatch-events": "3.501.0",
    "@aws-sdk/client-cloudwatch-logs": "3.501.0",
    "@aws-sdk/credential-providers": "3.501.0",
    "@aws-sdk/client-rds": "3.501.0",
    "@aws-sdk/client-iam": "3.501.0",
    "@aws-sdk/client-codebuild": "3.501.0",
    "@aws-sdk/client-efs": "3.501.0",
    "@aws-sdk/client-elastic-load-balancing": "3.501.0",
    "@aws-sdk/client-elastic-load-balancing-v2": "3.501.0",
    "@aws-sdk/client-organizations": "3.501.0",
    "@aws-sdk/client-pricing": "3.501.0",
    "@aws-sdk/client-opensearch": "3.501.0",
    "@aws-sdk/client-ecs": "3.501.0",
    "@aws-sdk/client-sqs": "3.501.0",
    "@aws-sdk/client-ssm": "3.501.0",
    "@aws-sdk/client-cloudwatch": "3.501.0",
    "@aws-sdk/client-inspector2": "3.501.0",

Result

node s3-test.js 
{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'MPT7SQX9XDENWSYW',
    extendedRequestId: '1l4lsiTdAMwyL0+FwrhaySPf0cjFeODCcaMWzUUGEqs+EQJ2CXMFGc3z6hVbUgIIK0/2fa8a+UE=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Buckets: [
    { 
    .....

@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

@gilesvessey I'm unable to reproduce this. Is the below similar to your setup? Trying to minimize it to the smallest example. I can't test in us-gov-east-1, but there should be no difference in this part of the AWS SDK.

my setup:

# config
[profile targets-default]
source_profile = default
role_arn = arn:aws:iam::...:role/AdminTestRole
# credentials
[default]
aws_access_key_id=...
aws_secret_access_key=...
aws_session_token=...
import { STS } from "@aws-sdk/client-sts";

process.env.AWS_PROFILE = "targets-default";
process.env.AWS_REGION = "us-west-2";

const client = new STS({});
console.log(await client.getCallerIdentity({})); // httpStatusCode 200 OK, assumed-role/AdminTestRole

@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

@jonathon-mcnabb Yes, the token's issuer will correspond to the error message. Rather, and I'm sorry if this is too obvious a question, but do you have this provider in your AWS account? It would be under

IAM > Identity Providers and the name of the provider should be the same as the issuer from the token without the https:// part if it is OIDC.

@jonathon-mcnabb
Copy link
Author

Ah ok. Thanks for clarifying. Yup, this provider does appear to be in our AWS account.

@kuhe
Copy link
Contributor

kuhe commented Jan 31, 2024

Did an EKS redeployment happen that would subtly change the ID value of the provider, that could cause a mismatch?

@jonathon-mcnabb
Copy link
Author

jonathon-mcnabb commented Jan 31, 2024

@kuhe Hmm, not that I am aware of.

I was able to test the below code in us-gov-west-1 and did not have any issues. Same result with 3.504.0. It looks like there isn't a version 3.503.* for @aws-sdk/client-sts?

import { STS } from "@aws-sdk/client-sts";

process.env.AWS_PROFILE = "test-profile";
process.env.AWS_REGION = "us-gov-west-1";

const client = new STS({});
console.log(await client.getCallerIdentity({})); // httpStatusCode 200 OK, assumed-role/AWSReservedSSO_AdministratorAccess_.......

@jonathon-mcnabb
Copy link
Author

Attempting a rebuild of our service code using version 3.504.0 of the SDK to see if that resolves anything. Will post results here soon.

@jonathon-mcnabb
Copy link
Author

@kuhe, it appears the same issue is still occurring on version 3.504.0

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Feb 1, 2024
@gilesvessey
Copy link

@gilesvessey I'm unable to reproduce this. Is the below similar to your setup? Trying to minimize it to the smallest example. I can't test in us-gov-east-1, but there should be no difference in this part of the AWS SDK.

my setup:

# config
[profile targets-default]
source_profile = default
role_arn = arn:aws:iam::...:role/AdminTestRole
# credentials
[default]
aws_access_key_id=...
aws_secret_access_key=...
aws_session_token=...
import { STS } from "@aws-sdk/client-sts";

process.env.AWS_PROFILE = "targets-default";
process.env.AWS_REGION = "us-west-2";

const client = new STS({});
console.log(await client.getCallerIdentity({})); // httpStatusCode 200 OK, assumed-role/AdminTestRole

My credential configuration is different than that in a few ways,

  • I am using long lived access keys, so no session token (we have an IAM user created for CI purposes)
  • My default profile references a non-default profile in the source_profile field. Then, I am using the default profile in my tests. Your example has it the other way around
  • I am using only one AWS environment variable - AWS_SDK_LOAD_CONFIG=1. I am relying entirely on the default profile configuration in the config/ credentials files. So, perhaps try setting the region in your config file and don't set any AWS_ environment variables other than AWS_SDK_LOAD_CONFIG=1
  • The role I am assuming requires an external_id

I am going to test the same configuration in us-east-1 to see what happens. Will report back

@kuhe
Copy link
Contributor

kuhe commented Feb 1, 2024

@gilesvessey let's move our discussion to #5755, leave this one for updates to the original problem.

@jonathon-mcnabb
Copy link
Author

hey @kuhe,

Is there any other information I could provide that may be helpful? I would also be happy to jump on a call to dive into things a bit deeper.

let me know!

@kuhe
Copy link
Contributor

kuhe commented Feb 1, 2024

I would be ok with getting on a call, but I can't post any contact information in this public forum. Could you send an invite to some form of communication? If you don't want to post it here, contact the AWS SDK for JavaScript team via your AWS account manager.

@jonathon-mcnabb
Copy link
Author

jonathon-mcnabb commented Feb 1, 2024

@kuhe Sounds good. I reached out to our account manager to set something up.

In the meantime, I ran a debugger on our source code and I found a clue:

In the "/usr/src/app/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js" file, there is an "endpointParams" variable that is returned as part of the "resolveParams" function.

When inspecting this value, it had a region set to "us-east-1". When I manually changed this value to my region "us-gov-west-1", we no longer got the "InvalidIdentityTokenException" error and the connection to AWS was successful. I have confirmed that the AWS_REGION env variable is set to us-gov-west-1.

{
  UseGlobalEndpoint: false,
  UseFIPS: false,
  Endpoint: undefined,
  Region: "us-east-1",
  UseDualStack: false,
}

Whats interesting is that this only occurs when processing the "AssumeRoleWithWebIdenityCommand".

It appears that the region is picked up by the GetObjectCommand, where this is the value of endpointParams:

{
  ForcePathStyle: false,
  UseArnRegion: false,
  DisableMultiRegionAccessPoints: false,
  Accelerate: false,
  DisableS3ExpressSessionAuth: false,
  UseGlobalEndpoint: false,
  UseFIPS: false,
  Endpoint: undefined,
  Region: "us-gov-west-1",
  UseDualStack: false,
  Bucket: "[REDACTED]",
  Key: "[REDACTED]",
}

@jonathon-mcnabb
Copy link
Author

jonathon-mcnabb commented Feb 1, 2024

@kuhe
Copy link
Contributor

kuhe commented Feb 1, 2024

Ok, thanks for the clue on where to look. I'm not sure how this managed to work previously, but I think a config change could be needed going forward.

With credential providers that involve an AWS service like STS, there's the concept of an outer and inner client.

If you use DynamoDB for example, that would be the outer client, and the credentials are supplied from an inner STS client. There are several potential places to set the region.

// This sets it on a global level for all AWS SDKs
// However, this does not set the region for any inner STS credential clients.
// As far as I can tell, this has always been the case.
process.env.AWS_REGION = 'us-gov-west-1'; 
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';

// Initializing with no args uses the default credential chain, 
// which may involve one of several inner clients (STS, SSO, etc.)
new DynamoDBClient({});
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';

// this is the explicit version of the default init, same as above.
new DynamoDBClient({
  credentials: fromNodeProviderChain({}) 
});

// you can also use a specific credential provider instead of the chain.
import { fromTokenFile } from '@aws-sdk/credential-providers';

new DynamoDBClient({
  // no inputs needed if environment variables are set for token usage 
  credentials: fromTokenFile({}) 
});

Both the outer and inner client's regions can be configured:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { fromNodeProviderChain } from '@aws-sdk/credential-providers';

new DynamoDBClient({
  region: 'us-gov-west-1', // outer client region
  credentials: fromNodeProviderChain({
    clientConfig: { 
      region: 'us-gov-west-1', // inner client region
    }
  }) 
});

Please try setting the inner client config region as above, giving it either the desired region or setting it to process.env.AWS_REGION.

@kuhe
Copy link
Contributor

kuhe commented Feb 1, 2024

Part of the reason for this input redundancy is that our credential providers fromXYZ({ clientConfig: ... }) return a standalone function that can be used to retrieve credentials. It is not aware of what client it is being fed to.

For SDK initialization that isn't in the default commercial regions, I think you end up needing to configure the region in two places.

@kuhe kuhe changed the title Urgent: InvalidIdentityTokenException with AWS SDK in Version 3.503.0 causes immediate failure Urgent: InvalidIdentityTokenException with AWS SDK in Version 3.503.0 causes immediate failure (govcloud) Feb 1, 2024
@kuhe kuhe added the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Feb 1, 2024
@jonathon-mcnabb
Copy link
Author

hmm, ok. It will be challenging for us to update in all of the locations this may need to occur.

Wouldn't it be expected that the AWS_REGION env variable is sourced for both the outer client AND the inner client?

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to \"closing-soon\" in 7 days. label Feb 2, 2024
@kuhe
Copy link
Contributor

kuhe commented Feb 2, 2024

There is pre-existing logic in place to default the inner STS clients to us-east-1 instead of the environment region, I believe because that serves as the global commercial region.

I will investigate whether I can change this to more logically handle other partitions.

@kuhe kuhe added queued This issues is on the AWS team's backlog pending-release This issue will be fixed by an approved PR that hasn't been released yet. and removed investigating Issue is being investigated and/or work is in progress to resolve the issue. queued This issues is on the AWS team's backlog labels Feb 3, 2024
@kuhe
Copy link
Contributor

kuhe commented Feb 12, 2024

This should be fixed as of this release: https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.511.0

When initializing a client with no explicit credentials (and therefore using the defaultCredentialProvider) like this:

new Client({
  region: "govcloud-region"
});

If there is an STS assume role command involved, this inner STS client's region will default to the outer client's resolved region if this region is not in the AWS default partition.

@kuhe kuhe added closing-soon This issue will automatically close in 4 days unless further comments are made. and removed pending-release This issue will be fixed by an approved PR that hasn't been released yet. labels Feb 12, 2024
@Grmiade
Copy link

Grmiade commented Feb 15, 2024

This issue seems to be strangely closed to #5799

@github-actions github-actions bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 16, 2024
@kuhe kuhe closed this as completed Feb 23, 2024
Copy link

github-actions bot commented Mar 9, 2024

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue is a bug. p1 This is a high priority issue
Projects
None yet
Development

No branches or pull requests

5 participants