OperationNotPermitted: You are not allowed to manage 'ela-attach' attachments

[heinosasshallik]

Looks like aws-nuke fails when there is a network interface that needs to be detached. From stderr:

level=error msg="OperationNotPermitted: You are not allowed to manage \'ela-attach\' attachments

Full logs (aws-nuke version 2.19.0):

• stderr:
  awsnukefailstderr.log
• stdout:
  awsnukefailstdout.log

Note: many of the errors in the stderr logs are not actually fatal. Refer to #1231

Tested against:

• multiple separate accounts that have network attachments. Aws-nuke failed and contained the above error in the logs
• one account that did not have a network attachment. Aws-nuke completed successfully (status code 0)

---

Manual verification

I initially thought the problem was related to the invalid credentials error messages or perhaps the service control policy error messages. However, that does not seem to be the case:

• I manually verified that the credentials given to aws-nuke are correct and work properly
• I manually verified the SCPs and they are the same as they have always been (errors started popping up lately, otherwise the same system has been working fine for over a year).

---

Version upgrade testing

I thought perhaps upgrading to the latest version would fix the issue. I upgraded to version 2.25.0, but I still get an error (process exit code 255). Except this time, I don't see the ela-attach-related error message in the logs. I'm not exactly sure what's going wrong here.

Logs:

• stderr:
  awsnuke225failstderr.log
• stdout:
  awsnuke225failstdout.log

[heinosasshallik]

Looking through the AWS account manually, I can confirm that 2.25.0 successfully deleted the network interface. So there is some sort of other problem now

[heinosasshallik]

There are some OpenSearch errors at the bottom of the 2.25 stderr logs. I manually went through all the sidebar options in Amazon OpenSearch. Everything seems to be empty.

[heinosasshallik]

I tested version 2.25 against an account that 2.19 was able to nuke successfully and 2.25 failed to nuke that account. The logs look the same, with stderr ending with OpenSearch errors

[heinosasshallik]

Using the aws-nuke option --exclude OSPackage fixed the error. But I think there's some sort of bug with aws-nuke here because as far as I can tell, I haven't done anything with OpenSearch and there shouldn't be anything aws-nuke couldn't delete here

[ekristen]

There seems to be a mess of errors happening. First and foremost, there seems to be issues with the credentials you are using. There are a lot of sts authentication errors or invalid security token happening. How are you authenticating?

The 255 error looks to be that an OpenSearch Package (OSPackage) is trying to be deleted the default packages, this has been fixed in my fork and in this code base, but a new version of 2.x has not been released with the fix.

Excluding OSPackage is a workaround for your exit 255 for now.

[heinosasshallik]

I'm authenticating to an OrganizationAccountAccessRole via temporary access credentials (access key, secret key, session token).

[heinosasshallik]

I'm not really sure what the credential failures could be related to. Perhaps due to credential propagation?

Then again, perhaps not. As you can see from the stderr logs, the first error that is shown is an AccessDenied error on iam:GetRole that occurs because of a service control policy. Assuming all credentials get validated against STS, it shouldn't be possible for a credential to be valid at one time (leading to the iam:GetRole failure) and then invalid later on (the expiry date is 10 hours so there's no way it would have expired that fast).

I think the iam:GetRole error is also a bug, by the way. I had filtered out the OrganizationAccountAccessRole in the aws-nuke config, but for some reason it's still trying to perform iam:GetRole against that user. Luckily it doesn't cause a failure in aws-nuke, so I've just been ignoring it

[ekristen]

So are you using AWS keys with an assume role to OrganizationAccountAccessRole? Since you mentioned SCP it sounds like you have SCPs that are blocking actions, this would account for a lot of the failures. Most likely SCPs blocking specific actions in specific regions.

[ekristen]

Yes, filtering out of the OrganizationAccountAccessRole is standard, it is not done by default.

[heinosasshallik]

Yes, filtering out of the OrganizationAccountAccessRole is standard, it is not done by default.

I'm filtering out OrganizationAccountAccessRole like this:

    config = {
        "regions": regions,
        "account-blocklist": [parent_account_id],
        "accounts": {
            target_account_id: {
                "filters": {
                    "IAMRole": ["OrganizationAccountAccessRole"],
                }
            }
        },
    }

Since you mentioned SCP it sounds like you have SCPs that are blocking actions, this would account for a lot of the failures

I don't know. The SCP only stops anything from messing with OrganizationAccountAccessRole and the aws-nuke config filters out that role anyways.

This is what the SCP does:

{
  "Statement": [
    {
      "Effect": "Deny",
      "Resource": "arn:aws:iam::*:role/OrganizationAccountAccessRole",
      "Sid": "ProtectAccessRole"
    },
  ],
  "Version": "2012-10-17"
}

There's also a condition about when not to block access to OrganizationAccountAccessRole and a statement that allows all other actions, but you can see that it's pretty simplistic

[ekristen]

So filtering out does not stop the tool from querying. We have no insight into SCPs so it will try and discover all roles, and get details and then filter them from being targeted.