Container hang when Okta token expires

[Gary-Armstrong]

Description
My Okta token expires after 60 minutes. When it expires and I use aws cli I see an error:

╰─ aws s3 ls s3://<bucketname>

An error occurred (ExpiredToken) when calling the ListObjectsV2 operation: The provided token has expired.

The container hangs in this case and does not display an error. Ideally it would halt and report an error.

Environment

• OS: Docker 3.0.3 - macos 11.1
• driftctl version: 0.2.3 - cloudskiff/driftctl 0379037aea34

How to reproduce
Use Okta SSO to generate ~/.aws/credentials
Let credentials expire

docker run -t --rm \
  -v ~/.aws:/home/.aws:ro \
  -v $(pwd)/terraform.tfstate:/app/terraform.tfstate:ro \
  -v ~/.driftctl:/app/.driftctl \
  -e AWS_PROFILE=$AWS_PROFILE \
cloudskiff/driftctl scan

Possible Solution
I think simply halting and displaying an error is correct.

[Gary-Armstrong]

While I know my token had expired, I wasn't using -e LOG_LEVEL=debug so it is possible I'm not seeing this correctly. In particular the AWS account I'm using has "a lot" of resources and even though I don't know how it could make progress with an expired token, I should test this again later using debug output.

[sjourdan]

thanks! don't hesitate to also use the --error-reporting flag, so we can get the full trace!

We recently worked with Vault revoking IAM keys (through IAM roles), and we encountered simple 403s - that was easy to handle.

We'll try to simulate a credential expiration like yours asap: your trace will be very helpful when you can (--error-reporting)!

[eliecharra]

While I know my token had expired, I wasn't using -e LOG_LEVEL=debug so it is possible I'm not seeing this correctly. In particular the AWS account I'm using has "a lot" of resources and even though I don't know how it could make progress with an expired token, I should test this again later using debug output.

Hi @Gary-Armstrong, thanks for all theses precious bug reports 👌🏻

For long running operations, driftctl remain silent at the moment. So you definitively may want to run it with -e LOG_LEVEL=debug to follow live execution. For this kind of token expiration we expect resource enumeration to fail with a 403, so maybe there is something bad on our side with this usecase. Keep in mind that --error-reporting will no report anything if no error are encountered. We need to try to reproduce this case on our side.

[Gary-Armstrong]

Better test.

╰─ aws s3 ls s3://<bucketname>

An error occurred (ExpiredToken) when calling the ListObjectsV2 operation: The provided token has expired.

Immediate error from aws cli.

╰─ terraform plan

Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: ExpiredToken: The security token included in the request is expired
	status code: 403, request id: 2e5b6d60-94bb-41f3-8e47-fae3791c1c43

Immediate error from terraform.

╰─ docker run -t --rm \
  -v ~/.aws:/home/.aws:ro \
  -v $(pwd):/app:ro \
  -v ~/.driftctl:/home/.driftctl \
  -e AWS_PROFILE=$AWS_PROFILE \
  -e LOG_LEVEL=debug \
cloudskiff/driftctl --error-reporting --from tfstate+s3://<bucketname>/<statefile> scan
DEBU[0000] Enabled error reporting
DEBU[0000] Found existing aws provider path=/home/.driftctl/plugins/linux_amd64/terraform-provider-aws_v3.19.0_x5
DEBU[0000] Starting new provider region=us-east-1
DEBU[0000] Starting aws provider GRPC client region=us-east-1

Remains at this point for four+ minutes in this experiment.

On CTRL-C:

WARN[0338] Detected interrupt during terraform provider configuration, cleanup ...

Immediate response.

[sjourdan]

Thank you @Gary-Armstrong this is very helpful for us to replicate this bug on our side!

[wbeuil]

Hi @Gary-Armstrong, I tried today to replicate your issue but unfortunately I couldn't :(

This is what I had for AWS CLI and terraform plan

And below for driftctl inside a docker container

Can you try again with our latest version v0.3.1 ?

[Gary-Armstrong]

This time I just let it go while I went to a meeting, and eventually forgot about it. After 54 minutes it produced an error.

╰─ docker run -t --rm \
  -v ~/.aws:/home/.aws:ro \
  -v $(pwd):/app:ro \
  -v ~/.driftctl:/home/.driftctl \
  -e AWS_PROFILE=$AWS_PROFILE \
  -e LOG_LEVEL=debug \
cloudskiff/driftctl --error-reporting --from tfstate+s3://BUCKET/KEY scan
DEBU[0000] Enabled error reporting
DEBU[0000] Found existing aws provider path=/home/.driftctl/plugins/linux_amd64/terraform-provider-aws_v3.19.0_x5
DEBU[0000] Starting new provider region=us-east-1
DEBU[0000] Starting aws provider GRPC client region=us-east-1

Time lapse occurs here.

Usage: driftctl scan [flags]

FLAGS:
      --filter string   JMESPath expression to filter on
                        Examples :
                          - Type == 'aws_s3_bucket' (will filter only s3 buckets)
                          - Type =='aws_s3_bucket && Id != 'my_bucket' (excludes s3 bucket 'my_bucket')
                          - Attr.Tags.Terraform == 'true' (include only resources that have Tag Terraform equal to 'true')

  -f, --from strings    IaC sources, by default try to find local terraform.tfstate file
                        Accepted schemes are: tfstate://,tfstate+s3://
                         (default [tfstate://terraform.tfstate])
  -o, --output string   Output format, by default it will write to the console
                        Accepted formats are: console://,json://PATH/TO/FILE.json
                         (default "console://")
  -t, --to string       Cloud provider source
                        Accepted values are: aws+tf
                         (default "aws+tf")

INHERITED FLAGS:
      --error-reporting    Enable error reporting.
                           WARNING: may leak sensitive data
  -h, --help               Display help for command
      --no-version-check   Disable the version check

error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: ExpiredToken: The security token included in the request is expired
	status code: 403, request id: b277667f-6ecc-4532-9d7e-c980d6616101
Sending error report ... done, thank you ❤️

I agree generally that it reports the error but I'd prefer it not take 54 minutes to do so.

[Gary-Armstrong]

Unsure if this is important, but:
driftctl v0.3.1 , image ee3a96ce6127
Docker 3.1.0(51484)
macos 11.2

[eliecharra]

Unsure if this is important, but:
driftctl v0.3.1 , image ee3a96ce6127
Docker 3.1.0(51484)
macos 11.2

Thanks a lot, as you are using error reporting, we got all required analytics on our side. We'll investigate this thanks a lot 🙏🏻

[sentry-io]

Sentry issue: DRIFTCTL-D

[wbeuil]

Hey @Gary-Armstrong, we finally found what is the problem here. I will try to explain it as much as I can.

Internally we use an older version of aws-sdk and an older version of the terraform-provider-aws (which also used an outdated aws-sdk version). Both of them don't support the use of SSO named profile as per AWS documentation. This was added in a recent PR here on aws-sdk repo.

That's why we (and probably you) need to use the key pair named profile where you need to pass aws_access_key_id, aws_secret_access_key and aws_session_token. Please correct me if I'm wrong here.

The problem occurs when your token expires: we don't renew the expired token and thus it hangs in an endless loop as described in this terraform issue.

Indeed we both, driftctl and terraform, have the same problem if you use the exact same setup:

• Okta SSO setup
• Named profile with access keys

# ~/.aws/credentials

[okta]
aws_access_key_id=access_key
aws_secret_access_key=secret
aws_session_token=token

# ~/terraform-repo/provider.tf

provider "aws" {
  region  = "eu-west-3"
  profile = "okta"
}

The first time you try it should work since the token is not expired yet.

The second time here's what you should have:

$ AWS_PROFILE=okta aws s3 ls
An error occurred (ExpiredToken) when calling the ListBuckets operation: The provided token has expired.

$ AWS_PROFILE=okta driftctl scan
DEBU[0000] New provider library created
DEBU[0000] Found existing aws provider path=/Users/wbeuil/.driftctl/plugins/darwin_amd64/terraform-provider-aws_v3.19.0_x5
DEBU[0000] Starting new provider region=eu-west-3
DEBU[0000] Starting aws provider GRPC client region=eu-west-3
# it hangs from now on

$ terraform plan
# it hangs from now on

To conclude, for now on this version of driftctl there's not much we can do since we rely on bugged version.

BUT I will open an issue to bump aws-sdk to at least v1.37.0 and terraform-provider-aws to at least v3.26.0 where SSO named profile (e.g. find below an example config) is supported and should take into account automatic renew of STS token as explained in the doc with this sentence: As long as you signed in to AWS SSO and those cached credentials are not expired, the AWS CLI automatically renews expired AWS temporary credentials when needed. However, if your AWS SSO credentials expire, you must explicitly renew them by logging in to your AWS SSO account again.

# ~/.aws/config

[profile sso]
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789011
sso_role_name = readOnly
region = eu-west-3

[Gary-Armstrong]

I am indeed using a named profile, although I didn't directly specify it you can see I have $AWS_PROFILE in the docker command.
This all seems correct, more or less. Sounds like my TF and awscli return immediately since they are up to date, while driftctl is using an older SDK.

In my case, when my credentials expire (one hour), I need to regenerate them myself because AWS CLI will not do it for me. We use a forked and modified https://github.com/oktadeveloper/okta-aws-cli-assume-role and the manual regen is part of my required workflow.

As I said, this all seems generally correct and I thank you for looking into it. I can always simply CTRL-C out of the driftctl execution and regenerate my credentials.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sjourdan]

Works with SSO support: #53

⚠️ use --tf-provider-version 3.26.0 minimum aws provider version (SSO support in AWS provider)

[GeraldCrescione]

Hey @Gary-Armstrong
this is a bit of a late answer, but we just released v0.12.0 with support for SSO authentications and this should actually fix your issue. Feel free to give it a try :)