provider/aws: IGW prevents VPC from changing CIDR

[radeksimko]

I understand it is probably a rare case when one needs to change CIDR of the whole VPC, but I believe this should be working without any errors:

resource "aws_vpc" "main" {
    cidr_block = "10.10.0.0/16"
}

resource "aws_internet_gateway" "gw" {
    vpc_id = "${aws_vpc.main.id}"
}

$ terraform plan

~ aws_internet_gateway.gw
    vpc_id: "vpc-9a868f2a" => "${aws_vpc.main.id}"

-/+ aws_vpc.main
    cidr_block:                "10.0.0.0/16" => "10.10.0.0/16" (forces new resource)
    default_network_acl_id:    "acl-7ac5d4ea" => "<computed>"
    default_security_group_id: "sg-a512ddb0" => "<computed>"
    dhcp_options_id:           "dopt-123a2f81" => "<computed>"
    enable_classiclink:        "false" => "0"
    enable_dns_hostnames:      "" => "<computed>"
    enable_dns_support:        "" => "<computed>"
    main_route_table_id:       "rtb-31230870" => "<computed>"

Plan: 1 to add, 1 to change, 1 to destroy.

$ terraform apply

aws_vpc.main: Refreshing state... (ID: vpc-9a868f2a)
aws_internet_gateway.gw: Refreshing state... (ID: igw-1a1110ba)
aws_vpc.main: Destroying...
Error applying plan:

1 error(s) occurred:

* aws_vpc.main: DependencyViolation: The vpc 'vpc-9a868f2a' has dependencies and cannot be deleted.
    status code: 400, request id:

I'm thinking easy the fix can be to just make aws_internet_gateway.vpc_id ForceNew: true.

[ocxo]

Ran into something similar today:

17:58:29 * aws_internet_gateway.gateway: InvalidParameterValue: Network vpc-75324b11 already has an internet gateway attached

[ElliotG]

+1, also ran into this.

[antoniobeyah]

👍

[desktophero]

👍

[morman]

👍

[radeksimko]

Just to add a possibly less obvious fact (and reason this issue stayed here and wasn't migrated away as part of the provider split) that emerged in our conversation over in hashicorp/terraform-provider-aws#1248

This bug affects all resources inside the VPC - aws_instance, aws_autoscaling_group, aws_db_instance, aws_eip, aws_elb, aws_alb etc. etc. Maybe half of all AWS resources can be attached to a VPC, so an exhaustive list would be very long.

[apparentlymart]

@radeksimko is it possible to move resources between VPCs without recreating them, assuming a situation where the VPCs are not being deleted/replaced?

Putting ForceNew on these various vpc_id attributes would make sense if that attribute can't be changed anyway (which I expect is true for some of them, but maybe not all?) but if any resource can currently be moved between VPC without recreating it then it'd be frustrating to have Terraform impose that additional constraint.

This issue feels like it's in a similar area as #8099. Ideally Terraform (with help from the provider) would be able to figure out on its own that when an aws_internet_gateway depends on a aws_vpc then the former must be replaced in order to replace the latter, but #8099 describes a compromise where those special relationships would need to be spelled out manually by the user.

A tricky part of this is that the replacement of the aws_internet_gateway (or any of these other resources in a similar situation) needs to be split up so that it can "straddle" the VPC replacement; in principle we can draw a graph that represents that, but I don't think we have any situations where that is true today:

• destroy aws_internet_gateway
• destroy aws_vpc
• create aws_vpc
• create aws_internet_gateway

[radeksimko]

Putting ForceNew on these various vpc_id attributes would make sense if that attribute can't be changed anyway (which I expect is true for some of them, but maybe not all?)

That's my understanding as well. I'm pretty sure EC2 instances and compute generally can't be moved between VPCs, neither subnets, but I can imagine moving route tables and IGWs. Admittedly I haven't tried this.

Either way the solution will involve some enhancements in the core (schema) prior to fixing it in the provider/resource code.

[Indigenuity]

I may be misunderstanding something, but I don't think this applies equally to ALL resources that depend on a VPC. Changing the vpc id for a subnet correctly says forces a new resource for vpc_id:

-/+ module.main.aws_subnet.private_app (new resource required)
      id:                               "subnet-REDACTED" => <computed> (forces new resource)
      arn:                              "arn:aws:ec2:us-east-1:REDACTED:subnet/subnet-REDACTED" => <computed>
      assign_ipv6_address_on_creation:  "false" => "false"
      availability_zone:                "us-east-1b" => "us-east-1b"
      cidr_block:                       "10.1.16.0/20" => "10.10.16.0/20" (forces new resource)
      ipv6_cidr_block:                  "" => <computed>
      ipv6_cidr_block_association_id:   "" => <computed>
      map_public_ip_on_launch:          "false" => "false"
      tags.%:                           "0" => "0"
      vpc_id:                           "vpc-REDACTED" => "${aws_vpc.main.id}" (forces new resource)

Another resource directly VPC dependent, an S3 gateway:

-/+ module.main.aws_vpc_endpoint.s3 (new resource required)
      id:                               "vpce-REDACTED" => <computed> (forces new resource)
      cidr_blocks.#:                    "2" => <computed>
      dns_entry.#:                      "0" => <computed>
      network_interface_ids.#:          "0" => <computed>
      policy:                           "{\"Statement\":[{\"Action\":\"*\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Resource\":\"*\"}],\"Version\":\"2008-10-17\"}" => <computed>
      prefix_list_id:                   "pl-REDACTED" => <computed>
      private_dns_enabled:              "false" => "false"
      route_table_ids.#:                "6" => <computed>
      security_group_ids.#:             "0" => <computed>
      service_name:                     "com.amazonaws.us-east-1.s3" => "com.amazonaws.us-east-1.s3"
      state:                            "available" => <computed>
      subnet_ids.#:                     "0" => <computed>
      vpc_endpoint_type:                "Gateway" => "Gateway"
      vpc_id:                           "vpc-REDACTED" => "${aws_vpc.main.id}" (forces new resource

But in the same changeset, the internet gateway:

  ~ module.main.aws_internet_gateway.main
      vpc_id:                           "vpc-REDACTED" => "${aws_vpc.main.id}"

Just seems to be inconsistent behavior by the provider

[teamterraform]

On the Terraform Core side of this problem, we now have #22094 as a description of the generalized problem (giving Terraform more information about relationships between objects), and so we're going to close this AWS-specific issue to consolidate over there, while also creating a record of the use-case. (A variant of this use-case with VPC subnets is already an example in that issue, though.)

It's possible that a change to ForceNew could still be warranted on the provider side to address this in the meantime, but the AWS provider is no longer developed in this repository and so any change of that sort would need to be made in the AWS provider repository.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.