Unable to update cluster_endpoint_private_access_cidrs

[wolstena]

I have issues

I'm submitting a...

• bug report
• feature request
• support request - read the FAQ first!
• kudos, thank you, warm fuzzy

What is the current behavior?

I am unable to update the cluster_endpoint_private_access_cidrs. The security group is initially created without an issue. However, when modify the definition of the cidr array it errors out. It is related to #hashicorp/terraform#2376. But, I have not been able to resolve the issues using the suggesting in the other ticket.

If this is a bug, how to reproduce? Please include a code sample if relevant.

Orginial Setting
cluster_endpoint_private_access_cidrs = ["104.145.4.34/32", "64.114.102.2/32", "172.21.224.0/24", "172.20.224.0/24", "172.21.226.0/24", "172.20.226.0/24"]
==> Updated Setting
cluster_endpoint_private_access_cidrs = ["104.145.4.34/32", "64.114.102.2/32", "172.21.224.0/24", "172.20.224.0/24", "172.21.226.0/24", "172.20.226.0/24", "10.142.8.0/21", "10.142.72.0/21"]

terragrunt plan

+/- resource "aws_security_group_rule" "cluster_private_access" {
      ~ cidr_blocks              = [ # forces replacement                                               
            "104.145.4.34/32",                  
            "64.114.102.2/32",
            "172.21.224.0/24",                                                                          
            "172.20.224.0/24",
            "172.21.226.0/24",
            "172.20.226.0/24",                  
          + "10.142.8.0/21",
          + "10.142.72.0/21",
        ]                                                                                               
        from_port                = 443          
      ~ id                       = "sgrule-559762067" -> (known after apply)                            
      - ipv6_cidr_blocks         = [] -> null
      - prefix_list_ids          = [] -> null
        protocol                 = "tcp"        
        security_group_id        = "sg-0573f2540cafdd5dc"                                               
        self                     = false
      + source_security_group_id = (known after apply)                                                  
        to_port                  = 443                                                                  
        type                     = "ingress"      
    }

terragrunt apply

module.eks.local_file.kubeconfig[0]: Creating...
module.eks.aws_security_group_rule.cluster_private_access[0]: Creating...
module.eks.local_file.kubeconfig[0]: Creation complete after 0s [id=cb0196e5e2b80a5763913bb450702afb8b04720e]

Error: [WARN] A duplicate Security Group rule was found on (sg-0573f2540cafdd5dc). This may be
a side effect of a now-fixed Terraform issue causing two security groups with
identical attributes but different source_security_group_ids to overwrite each
other in the state. See https://github.com/hashicorp/terraform/pull/2376 for more
information and instructions for recovery. Error message: the specified rule "peer: 104.145.4.34/32, TCP, from port: 443, to port: 443, ALLOW" already exists

  on .terraform/modules/eks/terraform-aws-eks-12.2.0/cluster.tf line 50, in resource "aws_security_group_rule" "cluster_private_access":
  50: resource "aws_security_group_rule" "cluster_private_access" {


Releasing state lock. This may take a few moments...
[terragrunt] 2020/08/19 09:21:06 Hit multiple errors:

What's the expected behavior?

Update the security group rules cidr blocks. I tried some of the suggestions in #hashicorp/terraform#2376 such as tainting the security groups with no luck.

Are you able to fix this problem and submit a PR? Link here if you have already.

No

Environment details

• Affected module version:
• OS:
• Terraform version:
  Terraform v0.12.29

Any other relevant info

Initializing provider plugins...

• Checking for available provider plugins...
• Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
• Downloading plugin for provider "local" (hashicorp/local) 1.4.0...
• Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
• Downloading plugin for provider "vault" (hashicorp/vault) 2.12.2...
• Downloading plugin for provider "kubernetes" (hashicorp/kubernetes) 1.12.0...
• Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
• Downloading plugin for provider "helm" (hashicorp/helm) 1.2.4...
• Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...

[dpiddockcmp]

For some reason terraform is performing a create then destroy of the security group rule. This won't work as the rules must be deleted then created. Why is Terraform doing this? There is no lifecycle set on the resource that would be causing this.

I managed to reproduce this. There's the single change to make:

+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.eks.aws_security_group_rule.cluster_private_access[0] must be replaced
+/- resource "aws_security_group_rule" "cluster_private_access" {
      ~ cidr_blocks              = [ # forces replacement
            "104.145.4.34/32",
          + "64.114.102.2/32",
        ]
        from_port                = 443
      ~ id                       = "sgrule-2274143143" -> (known after apply)
      - ipv6_cidr_blocks         = [] -> null
      - prefix_list_ids          = [] -> null
        protocol                 = "tcp"
        security_group_id        = "sg-00b5e19b67266ef64"
        self                     = false
      + source_security_group_id = (known after apply)
        to_port                  = 443
        type                     = "ingress"
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Happens under TF 0.12.29 and 0.13.0. With both aws 2.70.0 and 3.3.0. Upgrading isn't an easy escape. Will need more investigation.

[barryib]

At the end, here is what I did to get a consistency on this rule:

resource "aws_security_group_rule" "cluster_private_access" {
  for_each   = var.create_eks && var.cluster_endpoint_private_access ? toset(var.cluster_endpoint_private_access_cidrs) : []
  type        = "ingress"
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = [each.value]

  security_group_id = aws_eks_cluster.this[0].vpc_config[0].cluster_security_group_id
}

This will let Terraform manage each CIDR within its own SG rule, making updates and removals really easy. But the downside of this method is the AWS limit (I think AWS limit to 50 inbound rules per SG).

[dpiddockcmp]

The default 50 rule limit applies whether you're creating them in a single aws_security_group_rule or 50 single rules. There's a lot of API call overhead and terraform tree dancing involved with creating the rules individually like this though.

Would have been nice if hashicorp hadn't broken "working" plans

[CyrilPeponnet]

I have a related issue I think

  # module.eks.aws_security_group_rule.cluster_private_access[0] must be replaced
+/- resource "aws_security_group_rule" "cluster_private_access" {
      ~ cidr_blocks              = [ # forces replacement
          - "0.0.0.0/0",
          + "192.168.0.0/19",
        ]
        from_port                = 443
      ~ id                       = "sgrule-1234" -> (known after apply)
      - ipv6_cidr_blocks         = [] -> null
      - prefix_list_ids          = [] -> null
        protocol                 = "tcp"
        security_group_id        = "sg-01234"
        self                     = false
      + source_security_group_id = (known after apply)
        to_port                  = 443
        type                     = "ingress"
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes


Error: Provider produced inconsistent final plan

When expanding the plan for
module.eks.aws_security_group_rule.cluster_private_access[0] to include new
values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" changed the planned action from
CreateThenDelete to DeleteThenCreate.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Getting rid of the inbound rule manually obvisouly fixed it.

[barryib]

I still think that this the only way (at least for now) to make this change consistent. See also https://discuss.hashicorp.com/t/security-group-rules-destroyed-instead-of-added/7528/6.

There's a lot of API call overhead and terraform tree dancing involved with creating the rules individually like this though.

I don't think that there is a big difference in API calls, in both case, the provider loop over CIDR blocks and make an API call for each of them https://github.com/hashicorp/terraform-provider-aws/blob/master/aws/resource_aws_security_group_rule.go#L691

[barryib]

@CyrilPeponnet your issue is different from @wolstena's. The issue we're discussing here is about the Terraform AWS Provider trying to recreate a SG rule which already exists : Error message: the specified rule "peer: 104.145.4.34/32, TCP, from port: 443, to port: 443, ALLOW" already exists

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity since being marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[lisfo4ka]

Folks, please, look into #1549 which implements @barryib suggestions #984 (comment).
Our team also requires this fix and I believe it's quite a good way to resolve the issue.

[daroga0002]

#1549 was merged so this issue seems be solved

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.