terraform takes 5 minutes to create a aws_security_group_rule.

[sk-cool]

Terraform vesion v0.6.16

I have the following type of rules :

"egress29": {
        "from_port": "8140",
        "protocol": "tcp",
        "security_group_id": "${aws_security_group.sgallhostrules.id}",
        "to_port": "8140",
        "type": "egress"
      },
      "egress3": {
        "cidr_blocks": [
          "10.252.1.0/24"
        ],
        "from_port": "443",
        "protocol": "tcp",
        "security_group_id": "${aws_security_group.sgAPPhostrules.id}",
        "to_port": "443",
        "type": "egress"
      }

The sgAPPhostrules looks like below -

"sgAPPhostrules": {
        "description": "rules to allow connectivity from Load Balancers",
        "vpc_id": "${aws_vpc.vpc29db544c.id}",
        "egress": {
          "from_port": "0",
          "to_port": "0",
          "protocol": "-1",
          "cidr_blocks": [
            "0.0.0.0/0"
          ]
        }
      }

Command line output of the terraform apply command is as follows --

aws_security_group_rule.egress29: Creating...
  from_port:                "" => "8140"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-600cf506"
  self:                     "" => "false"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "8140"
  type:                     "" => "egress"
aws_security_group_rule.egress29: Still creating... (10s elapsed)
aws_security_group_rule.egress29: Still creating... (20s elapsed)
aws_security_group_rule.egress29: Still creating... (30s elapsed)
aws_security_group_rule.egress29: Still creating... (40s elapsed)
aws_security_group_rule.egress29: Still creating... (50s elapsed)
aws_security_group_rule.egress29: Still creating... (1m0s elapsed)
aws_security_group_rule.egress29: Still creating... (1m10s elapsed)
aws_security_group_rule.egress29: Still creating... (1m20s elapsed)
aws_security_group_rule.egress29: Still creating... (1m30s elapsed)
aws_security_group_rule.egress29: Still creating... (1m40s elapsed)
aws_security_group_rule.egress29: Still creating... (1m50s elapsed)
aws_security_group_rule.egress29: Still creating... (2m0s elapsed)
aws_security_group_rule.egress29: Still creating... (2m10s elapsed)
aws_security_group_rule.egress29: Still creating... (2m20s elapsed)
aws_security_group_rule.egress29: Still creating... (2m30s elapsed)
aws_security_group_rule.egress29: Still creating... (2m40s elapsed)
aws_security_group_rule.egress29: Still creating... (2m50s elapsed)
aws_security_group_rule.egress29: Still creating... (3m0s elapsed)
aws_security_group_rule.egress29: Still creating... (3m10s elapsed)
aws_security_group_rule.egress29: Still creating... (3m20s elapsed)
aws_security_group_rule.egress29: Still creating... (3m30s elapsed)
aws_security_group_rule.egress29: Still creating... (3m40s elapsed)
aws_security_group_rule.egress29: Still creating... (3m50s elapsed)
aws_security_group_rule.egress29: Still creating... (4m0s elapsed)
aws_security_group_rule.egress29: Still creating... (4m10s elapsed)
aws_security_group_rule.egress29: Still creating... (4m20s elapsed)
aws_security_group_rule.egress29: Still creating... (4m30s elapsed)
aws_security_group_rule.egress29: Still creating... (4m40s elapsed)
aws_security_group_rule.egress29: Still creating... (4m50s elapsed)
aws_security_group_rule.egress29: Still creating... (5m0s elapsed)
aws_security_group_rule.egress29: Creation complete

It takes 5 minutes to create one rule. We have more than 100 such rules. The other resource creation is happens comparatively fast enough.

Thanks,
-Satish

[catsby]

hey @sk-cool is this reproducible? Can you run this with TF_LOG=1 and let me know if the debug output reveals any warning or errors here?

[sk-cool]

Hi @catsby, Thanks for your reply.

Yes, this is reproducible. Please find attached logs.

Please see below command line output.

aws_security_group_rule.egress40: Still creating... (4m10s elapsed)
aws_security_group_rule.egress38: Still creating... (4m20s elapsed)
aws_security_group_rule.egress29: Still creating... (4m0s elapsed)
aws_security_group_rule.egress39: Still creating... (4m20s elapsed)
aws_security_group_rule.egress32: Still creating... (4m10s elapsed)
aws_security_group_rule.ingress52: Still creating... (4m30s elapsed)
aws_security_group_rule.ingress53: Still creating... (4m30s elapsed)
aws_security_group_rule.egress37: Still creating... (4m30s elapsed)
aws_security_group_rule.egress41: Still creating... (4m30s elapsed)
aws_security_group_rule.ingress54: Still creating... (4m30s elapsed)
aws_security_group_rule.egress40: Still creating... (4m20s elapsed)
aws_security_group_rule.egress38: Still creating... (4m30s elapsed)
aws_security_group_rule.egress29: Still creating... (4m10s elapsed)

tfdebug.zip

Meanwhile, I tried the following which worked good.

1. Made sure that the aws_security_groups are created first.
2. Added only 4 aws_security_group_rules.

This happened pretty fast. However If I have more aws_security_group_rules it takes a lot of time.

I hope this helps debug the problem better.

Thanks,
Satish

[sk-cool]

I enabled debug logs by setting - TF_LOG=DEBUG environment variable.

Thanks,
Satish

[rileyje]

Any updates on this issue? We're hitting this on v0.6.16 regularly as well. Let me know if I can provide more information.

[sk-cool]

I think the problem in this issue is following -

1. Terraform tries to create the security groups in AWS.
2. While it is trying create multiple groups in parallel, it is also fetching the status of the group to know whether it is successfully created or not.
3. For some security groups it takes a lot of time to get the status or it never gets the status back, however the security group is successfully created in the aws.
4. It gets in to an endless loop here.

When I tried running this with parallelism=1, then things work fine. However it is still slow.
Also, please note this is just a workaround :(.

Thanks,
Satish

[hingstarne]

Same issue still in 0.7.4

[rgynn]

Still the same in 0.7.6

Rules are created way faster, but the status doesn't seem to reach terraform.
Haven't tried the parallelism=1 workaround.

[aberrios85]

Also hitting up against this in v0.8.1

[acesaro]

I think I might have figured this out, at least for the case I was encountering. I was seeing the same 5 minute duration for applying many individual aws_security_group_rule resources, but then noticed that some of my CIDR blocks were changing after they were implemented in AWS. I analyzed it a bit and noticed that I had overlapping CIDR blocks (10.5.0.0/15 and 10.4.0.0/16 in this case) and after correcting those (by splitting 10.5.0.0/15 into the proper 10.5.0.0/16 and 10.6.0.0/16 blocks) the apply ran quickly again.

So, my suggestion to anyone running into this issue is to make sure you have correctly separated your CIDR blocks and they don't overlap (maybe by closing looking at what ends up in AWS compared to your Terraform definitions).

[grubernaut]

Fixed in #11809

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.