AWS Provider in 0.5.3- Can not tie multiple aws_security_group_rule to one security group #2294

zxjinn · 2015-06-09T21:27:19Z

Again, I just love Terraform.. I sing its praises to everyone I meet!
The second issue I found deals again with aws_security_group_rule. I can not get one security group to attach multiple rules (which point to other security groups) without throwing errors on subsequent applys.

File structure and contents

tree

$ tree
.
├── main.tf
└── terraform.tfvars
0 directories, 2 files

main.tf

variable "access_key" {}
variable "region" {}
variable "secret_key" {}

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "${var.region}"
}
resource "aws_vpc" "vpc" {
  cidr_block           = "172.16.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags {
    Name = "asgr2-vpc"
  }
}

resource "aws_security_group" "nat" {
  description = "nat security group"
  name        = "asgr2-nat"
  vpc_id      = "${aws_vpc.vpc.id}"
  tags {
    Name = "asgr2-nat"
  }
}
resource "aws_security_group" "admin" {
  description = "admin security group"
  name        = "asgr2-admin"
  vpc_id      = "${aws_vpc.vpc.id}"
  tags {
    Name = "asgr2-admin"
  }
}
resource "aws_security_group" "bastion" {
  description = "bastion security group"
  name        = "asgr2-bastion"
  vpc_id      = "${aws_vpc.vpc.id}"
  tags {
    Name = "asgr2-bastion"
  }
}

resource "aws_security_group_rule" "admin_ingress_nat" {
  from_port                = 0
  protocol                 = "-1"
  security_group_id        = "${aws_security_group.admin.id}"
  source_security_group_id = "${aws_security_group.nat.id}"
  to_port                  = 0
  type                     = "ingress"
}
resource "aws_security_group_rule" "admin_ingress_bastion" {
  from_port                = 0
  protocol                 = "-1"
  security_group_id        = "${aws_security_group.admin.id}"
  source_security_group_id = "${aws_security_group.bastion.id}"
  to_port                  = 0
  type                     = "ingress"
}

terraform.tfvars

access_key = "<actual_access_key_here>"
region = "us-west-2"
secret_key = "<actual_secret_key_here>"

Terraform version

Tested with both:

$ terraform --version
Terraform v0.5.3

$ terraform --version
Terraform v0.6.0-dev (2b9318651236807c29add41a9119c92f59bf58ba)

Problem

I have had a number of issues getting security groups to reference each other, this one in particular I can't seem to get around. First I create three security groups, nat, admin, bastion. Second I allow all traffic from nat to admin. Third, allow all traffic from bastion to admin. This applies correctly on the first Terraform apply, but all other subsequent applies do not work properly.

Commands ran and statefiles

Plan

$ terraform plan
Refreshing Terraform state prior to plan...


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ aws_security_group.admin
    description: "" => "admin security group"
    egress.#:    "" => "<computed>"
    ingress.#:   "" => "<computed>"
    name:        "" => "asgr2-admin"
    owner_id:    "" => "<computed>"
    tags.#:      "" => "1"
    tags.Name:   "" => "asgr2-admin"
    vpc_id:      "" => "${aws_vpc.vpc.id}"

+ aws_security_group.bastion
    description: "" => "bastion security group"
    egress.#:    "" => "<computed>"
    ingress.#:   "" => "<computed>"
    name:        "" => "asgr2-bastion"
    owner_id:    "" => "<computed>"
    tags.#:      "" => "1"
    tags.Name:   "" => "asgr2-bastion"
    vpc_id:      "" => "${aws_vpc.vpc.id}"

+ aws_security_group.nat
    description: "" => "nat security group"
    egress.#:    "" => "<computed>"
    ingress.#:   "" => "<computed>"
    name:        "" => "asgr2-nat"
    owner_id:    "" => "<computed>"
    tags.#:      "" => "1"
    tags.Name:   "" => "asgr2-nat"
    vpc_id:      "" => "${aws_vpc.vpc.id}"

+ aws_security_group_rule.admin_ingress_bastion
    from_port:                "" => "0"
    protocol:                 "" => "-1"
    security_group_id:        "" => "${aws_security_group.admin.id}"
    self:                     "" => "0"
    source_security_group_id: "" => "${aws_security_group.bastion.id}"
    to_port:                  "" => "0"
    type:                     "" => "ingress"

+ aws_security_group_rule.admin_ingress_nat
    from_port:                "" => "0"
    protocol:                 "" => "-1"
    security_group_id:        "" => "${aws_security_group.admin.id}"
    self:                     "" => "0"
    source_security_group_id: "" => "${aws_security_group.nat.id}"
    to_port:                  "" => "0"
    type:                     "" => "ingress"

+ aws_vpc.vpc
    cidr_block:                "" => "172.16.0.0/16"
    default_network_acl_id:    "" => "<computed>"
    default_security_group_id: "" => "<computed>"
    dhcp_options_id:           "" => "<computed>"
    enable_dns_hostnames:      "" => "1"
    enable_dns_support:        "" => "1"
    main_route_table_id:       "" => "<computed>"
    tags.#:                    "" => "1"
    tags.Name:                 "" => "asgr2-vpc"

Apply

$ terraform apply
aws_vpc.vpc: Creating...
  cidr_block:                "" => "172.16.0.0/16"
  default_network_acl_id:    "" => "<computed>"
  default_security_group_id: "" => "<computed>"
  dhcp_options_id:           "" => "<computed>"
  enable_dns_hostnames:      "" => "1"
  enable_dns_support:        "" => "1"
  main_route_table_id:       "" => "<computed>"
  tags.#:                    "" => "1"
  tags.Name:                 "" => "asgr2-vpc"
aws_vpc.vpc: Creation complete
aws_security_group.bastion: Creating...
  description: "" => "bastion security group"
  egress.#:    "" => "<computed>"
  ingress.#:   "" => "<computed>"
  name:        "" => "asgr2-bastion"
  owner_id:    "" => "<computed>"
  tags.#:      "" => "1"
  tags.Name:   "" => "asgr2-bastion"
  vpc_id:      "" => "vpc-9eef65fb"
aws_security_group.nat: Creating...
  description: "" => "nat security group"
  egress.#:    "" => "<computed>"
  ingress.#:   "" => "<computed>"
  name:        "" => "asgr2-nat"
  owner_id:    "" => "<computed>"
  tags.#:      "" => "1"
  tags.Name:   "" => "asgr2-nat"
  vpc_id:      "" => "vpc-9eef65fb"
aws_security_group.admin: Creating...
  description: "" => "admin security group"
  egress.#:    "" => "<computed>"
  ingress.#:   "" => "<computed>"
  name:        "" => "asgr2-admin"
  owner_id:    "" => "<computed>"
  tags.#:      "" => "1"
  tags.Name:   "" => "asgr2-admin"
  vpc_id:      "" => "vpc-9eef65fb"
aws_security_group.bastion: Creation complete
aws_security_group.nat: Creation complete
aws_security_group.admin: Creation complete
aws_security_group_rule.admin_ingress_bastion: Creating...
  from_port:                "" => "0"
  protocol:                 "" => "-1"
  security_group_id:        "" => "sg-49ed032d"
  self:                     "" => "0"
  source_security_group_id: "" => "sg-4fed032b"
  to_port:                  "" => "0"
  type:                     "" => "ingress"
aws_security_group_rule.admin_ingress_nat: Creating...
  from_port:                "" => "0"
  protocol:                 "" => "-1"
  security_group_id:        "" => "sg-49ed032d"
  self:                     "" => "0"
  source_security_group_id: "" => "sg-48ed032c"
  to_port:                  "" => "0"
  type:                     "" => "ingress"
aws_security_group_rule.admin_ingress_bastion: Creation complete
aws_security_group_rule.admin_ingress_nat: Creation complete

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Statefile, after first apply

{
    "version": 1,
    "serial": 1,
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {
                "aws_security_group.admin": {
                    "type": "aws_security_group",
                    "depends_on": [
                        "aws_vpc.vpc"
                    ],
                    "primary": {
                        "id": "sg-49ed032d",
                        "attributes": {
                            "description": "admin security group",
                            "egress.#": "0",
                            "id": "sg-49ed032d",
                            "ingress.#": "0",
                            "name": "asgr2-admin",
                            "owner_id": "317085423413",
                            "tags.#": "1",
                            "tags.Name": "asgr2-admin",
                            "vpc_id": "vpc-9eef65fb"
                        }
                    }
                },
                "aws_security_group.bastion": {
                    "type": "aws_security_group",
                    "depends_on": [
                        "aws_vpc.vpc"
                    ],
                    "primary": {
                        "id": "sg-4fed032b",
                        "attributes": {
                            "description": "bastion security group",
                            "egress.#": "0",
                            "id": "sg-4fed032b",
                            "ingress.#": "0",
                            "name": "asgr2-bastion",
                            "owner_id": "317085423413",
                            "tags.#": "1",
                            "tags.Name": "asgr2-bastion",
                            "vpc_id": "vpc-9eef65fb"
                        }
                    }
                },
                "aws_security_group.nat": {
                    "type": "aws_security_group",
                    "depends_on": [
                        "aws_vpc.vpc"
                    ],
                    "primary": {
                        "id": "sg-48ed032c",
                        "attributes": {
                            "description": "nat security group",
                            "egress.#": "0",
                            "id": "sg-48ed032c",
                            "ingress.#": "0",
                            "name": "asgr2-nat",
                            "owner_id": "317085423413",
                            "tags.#": "1",
                            "tags.Name": "asgr2-nat",
                            "vpc_id": "vpc-9eef65fb"
                        }
                    }
                },
                "aws_security_group_rule.admin_ingress_bastion": {
                    "type": "aws_security_group_rule",
                    "depends_on": [
                        "aws_security_group.admin",
                        "aws_security_group.bastion"
                    ],
                    "primary": {
                        "id": "sg-4235098228",
                        "attributes": {
                            "cidr_blocks.#": "0",
                            "from_port": "0",
                            "id": "sg-4235098228",
                            "protocol": "-1",
                            "security_group_id": "sg-49ed032d",
                            "self": "false",
                            "source_security_group_id": "sg-4fed032b",
                            "to_port": "0",
                            "type": "ingress"
                        }
                    }
                },
                "aws_security_group_rule.admin_ingress_nat": {
                    "type": "aws_security_group_rule",
                    "depends_on": [
                        "aws_security_group.admin",
                        "aws_security_group.nat"
                    ],
                    "primary": {
                        "id": "sg-4235098228",
                        "attributes": {
                            "cidr_blocks.#": "0",
                            "from_port": "0",
                            "id": "sg-4235098228",
                            "protocol": "-1",
                            "security_group_id": "sg-49ed032d",
                            "self": "false",
                            "source_security_group_id": "sg-4fed032b",
                            "to_port": "0",
                            "type": "ingress"
                        }
                    }
                },
                "aws_vpc.vpc": {
                    "type": "aws_vpc",
                    "primary": {
                        "id": "vpc-9eef65fb",
                        "attributes": {
                            "cidr_block": "172.16.0.0/16",
                            "default_network_acl_id": "acl-4edc4c2b",
                            "default_security_group_id": "sg-4ded0329",
                            "dhcp_options_id": "dopt-05080167",
                            "enable_dns_hostnames": "true",
                            "enable_dns_support": "true",
                            "id": "vpc-9eef65fb",
                            "main_route_table_id": "rtb-0e19876b",
                            "tags.#": "1",
                            "tags.Name": "asgr2-vpc"
                        }
                    }
                }
            }
        }
    ]
}

Note that the source_security_group_id for aws_security_group_rule.admin_ingress_bastion AND aws_security_group_rule.admin_ingress_nat both say the same id (sg-4fed032b), which is incorrect.
The one relating to bastion should say sg-4fed032b and the one relating to nat should say sg-48ed032c.
However, the rule was successfully applied. Screenshot from the AWS console directly after the first apply.

Apply, with no changes

$ terraform apply
aws_vpc.vpc: Refreshing state... (ID: vpc-9eef65fb)
aws_security_group.bastion: Refreshing state... (ID: sg-4fed032b)
aws_security_group.nat: Refreshing state... (ID: sg-48ed032c)
aws_security_group.admin: Refreshing state... (ID: sg-49ed032d)
aws_security_group_rule.admin_ingress_nat: Refreshing state... (ID: sg-4235098228)
aws_security_group_rule.admin_ingress_bastion: Refreshing state... (ID: sg-4235098228)
aws_security_group_rule.admin_ingress_nat: Destroying...
aws_security_group_rule.admin_ingress_nat: Destruction complete
aws_security_group_rule.admin_ingress_nat: Creating...
  from_port:                "" => "0"
  protocol:                 "" => "-1"
  security_group_id:        "" => "sg-49ed032d"
  self:                     "" => "0"
  source_security_group_id: "" => "sg-48ed032c"
  to_port:                  "" => "0"
  type:                     "" => "ingress"
aws_security_group_rule.admin_ingress_nat: Error: 1 error(s) occurred:

* Error authorizing security group rules rules: InvalidPermission.Duplicate: the specified rule "peer: sg-48ed032c, ALL, ALLOW" already exists
    status code: 400, request id: []
Error applying plan:

1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* Error authorizing security group rules rules: InvalidPermission.Duplicate: the specified rule "peer: sg-48ed032c, ALL, ALLOW" already exists
    status code: 400, request id: []

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Statefile diff, after failed apply

--- terraform.tfstate.backup Tue Jun  9 15:23:41 2015
+++ terraform.tfstate Tue Jun  9 15:23:41 2015
@@ -1,6 +1,6 @@
 {
     "version": 1,
-    "serial": 1,
+    "serial": 2,
     "modules": [
         {
             "path": [
@@ -89,27 +89,6 @@
                         }
                     }
                 },
-                "aws_security_group_rule.admin_ingress_nat": {
-                    "type": "aws_security_group_rule",
-                    "depends_on": [
-                        "aws_security_group.admin",
-                        "aws_security_group.nat"
-                    ],
-                    "primary": {
-                        "id": "sg-4235098228",
-                        "attributes": {
-                            "cidr_blocks.#": "0",
-                            "from_port": "0",
-                            "id": "sg-4235098228",
-                            "protocol": "-1",
-                            "security_group_id": "sg-49ed032d",
-                            "self": "false",
-                            "source_security_group_id": "sg-4fed032b",
-                            "to_port": "0",
-                            "type": "ingress"
-                        }
-                    }
-                },
                 "aws_vpc.vpc": {
                     "type": "aws_vpc",
                     "primary": {

I hope that wasn't TMI!
This might or might not relate to the other issue I just reported, #2291.

The text was updated successfully, but these errors were encountered:

zxjinn · 2015-06-09T21:30:44Z

Additionally, sometimes the subsequent applys throw many InvalidPermission.Duplicate errors, and sometimes it only throws one. Not sure what's going on there.

jszwedko · 2015-06-11T21:04:39Z

I think this is an issue with the fact that the IPPermission hashing in https://github.com/hashicorp/terraform/blob/master/builtin/providers/aws/resource_aws_security_group_rule.go#L266 only considers CIDR blocks and not other security groups as sources. I'm working on a fix now.

jszwedko · 2015-06-11T21:04:56Z

Basically you get different rules hashing to the same thing.

jszwedko · 2015-06-11T22:00:08Z

By the way this is an awesome issue report @zxjinn

zxjinn · 2015-06-11T22:59:26Z

Thanks @jszwedko 😄 I figure the people neck-deep in Go, doing the hard work, deserve as much help as they can get.

catsby · 2015-06-16T19:59:22Z

I opened #2376 (includes #2318) to address this. We'll review and update it as needed, and hopefully get it merged soon

Thanks for the extremely detailed report, btw 😄

catsby · 2015-06-19T16:51:10Z

Merged #2376 , thank you @zxjinn and @jszwedko !

ghost · 2020-05-02T02:03:56Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

zxjinn mentioned this issue Jun 9, 2015

AWS Provider in 0.5.3- Can not taint aws_security_group_rule #2291

Closed

catsby added bug provider/aws labels Jun 10, 2015

jszwedko mentioned this issue Jun 11, 2015

Consider security group rules with source security groups when hashing #2318

Closed

catsby mentioned this issue Jun 16, 2015

Consider security groups with source security groups when hashing #2376

Merged

catsby closed this as completed in #2376 Jun 19, 2015

ghost locked and limited conversation to collaborators May 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS Provider in 0.5.3- Can not tie multiple aws_security_group_rule to one security group #2294

AWS Provider in 0.5.3- Can not tie multiple aws_security_group_rule to one security group #2294

zxjinn commented Jun 9, 2015

zxjinn commented Jun 9, 2015

jszwedko commented Jun 11, 2015

jszwedko commented Jun 11, 2015

jszwedko commented Jun 11, 2015

zxjinn commented Jun 11, 2015

catsby commented Jun 16, 2015

catsby commented Jun 19, 2015

ghost commented May 2, 2020

AWS Provider in 0.5.3- Can not tie multiple aws_security_group_rule to one security group #2294

AWS Provider in 0.5.3- Can not tie multiple aws_security_group_rule to one security group #2294

Comments

zxjinn commented Jun 9, 2015

File structure and contents

tree

main.tf

terraform.tfvars

Terraform version

Problem

Commands ran and statefiles

Plan

Apply

Statefile, after first apply

Apply, with no changes

Statefile diff, after failed apply

zxjinn commented Jun 9, 2015

jszwedko commented Jun 11, 2015

jszwedko commented Jun 11, 2015

jszwedko commented Jun 11, 2015

zxjinn commented Jun 11, 2015

catsby commented Jun 16, 2015

catsby commented Jun 19, 2015

ghost commented May 2, 2020