Upgrade from 0.3.5->0.3.6 forces recreation of new aws_instances with block_storage #859

davedash · 2015-01-23T19:03:14Z

Almost all my EC2 instances need to be recreated when I run plan:

resource "aws_instance" "logstash" {
  ami = "${var.amis.ubuntu_ebs}"
  instance_type = "m3.xlarge"
  tags {
    Name = "logstash"
    env = "infra"
    role = "infra-logstash"
  }
  ebs_optimized = true
  security_groups = [
    "${aws_security_group.infra.id}",
    "${aws_security_group.logstash.id}"
  ]

  subnet_id = "${aws_subnet.infra.id}"
  iam_instance_profile = "infra-logstash"

  block_device {
    encrypted = true
    device_name = "/dev/sdf"
    volume_type = "gp2"
    volume_size = 50
  }
}

plan output:

-/+ aws_instance.logstash
    ami:                                           "ami-9eaa1cf6" => "ami-9eaa1cf6"
    availability_zone:                             "us-east-1c" => "<computed>"
    block_device.#:                                "2" => "1"
    block_device.2577914260.delete_on_termination: "" => "1" (forces new resource)
    block_device.2577914260.device_name:           "" => "/dev/sdf" (forces new resource)
    block_device.2577914260.encrypted:             "" => "1" (forces new resource)
    block_device.2577914260.snapshot_id:           "" => "<computed>" (forces new resource)
    block_device.2577914260.virtual_name:          "" => "" (forces new resource)
    block_device.2577914260.volume_size:           "" => "50" (forces new resource)
    block_device.2577914260.volume_type:           "" => "gp2" (forces new resource)
    ebs_optimized:                                 "true" => "1"
    iam_instance_profile:                          "infra-logstash" => "infra-logstash"
    instance_type:                                 "m3.xlarge" => "m3.xlarge"
    key_name:                                      "davedash" => "davedash"
    subnet_id:                                     "subnet-b1c719e8" => "subnet-b1c719e8"
    tags.#:                                        "" => "3"
    tags.Name:                                     "logstash" => "logstash"
    tags.env:                                      "infra" => "infra"
    tags.role:                                     "infra-logstash" => "infra-logstash"
    tenancy:                                       "default" => "<computed>"

tfstate:

                "aws_instance.logstash": {
                    "type": "aws_instance",
                    "depends_on": [
                        "aws_security_group.infra",
                        "aws_security_group.logstash",
                        "aws_subnet.infra"
                    ],
                    "primary": {
                        "id": "i-aecc5c42",
                        "attributes": {
                            "ami": "ami-9eaa1cf6",
                            "availability_zone": "us-east-1c",
                            "block_device.#": "2",
                            "block_device.2382792209.delete_on_termination": "true",
                            "block_device.2382792209.device_name": "/dev/sdf",
                            "block_device.2382792209.encrypted": "true",
                            "block_device.2382792209.snapshot_id": "",
                            "block_device.2382792209.virtual_name": "",
                            "block_device.2382792209.volume_size": "50",
                            "block_device.2382792209.volume_type": "gp2",
                            "block_device.2936417914.delete_on_termination": "true",
                            "block_device.2936417914.device_name": "/dev/sda1",
                            "block_device.2936417914.encrypted": "false",
                            "block_device.2936417914.snapshot_id": "snap-1f806dbb",
                            "block_device.2936417914.virtual_name": "",
                            "block_device.2936417914.volume_size": "8",
                            "block_device.2936417914.volume_type": "gp2",
                            "ebs_optimized": "true",
                            "iam_instance_profile": "infra-logstash",
                            "id": "i-aecc5c42",
                            "instance_type": "m3.xlarge",
                            "public_dns": "",
                            "public_ip": "",
                            "security_groups.#": "2",
                            "security_groups.1548674668": "sg-0d5aad69",
                            "security_groups.3312015480": "sg-a22dc2c6",
                            "subnet_id": "subnet-b1c719e8",
                            "tags.Name": "logstash",
                            "tags.env": "infra",
                            "tags.role": "infra-logstash",
                            "tenancy": "default"
                        }
                    }
                },

It's a little hard to see why a new resource is being forced here. Usually I can adjust something in either the tfstate, or alter the block storage stanza to get this to work.

The text was updated successfully, but these errors were encountered:

davedash · 2015-01-23T19:07:51Z

I should note that these are volumes with an EBS root volume as well, which could explain:

    block_device.#:                                "2" => "1"

davedash · 2015-01-23T19:09:19Z

Also I tried the trick in #824 to add:

delete_on_termination = true

and that did not help

mitchellh · 2015-01-23T20:47:24Z

Hm, this is going to be hard to debug without a more complete state file. Can you show the complete state of the instance minus perhaps the DNS?

davedash · 2015-01-23T20:52:07Z

Sorry about that @mitchellh. I added more back context of the tfstate in the original issue

mitchellh · 2015-01-23T20:55:24Z

Hm, based on the state it looks like it might be doing the right thing? Your config has one block device but your state has two. Can you explain that in more detail?

davedash · 2015-01-23T20:56:41Z

The state is listing the root volume (which is EBS) in addition to the attached non-root volume I defined in the .tf file.

mitchellh · 2015-01-23T20:57:08Z

Got it that's the bug then. We should probably ignore the root volume...

patricklucas · 2015-01-24T02:07:33Z

We hit this too, even starting clean from 0.3.6. The symptom was that Terraform wanted to recreate any aws_instance resources with block device mappings on every apply.

I'm downgrading to 0.3.5 in the mean time.

knuckolls · 2015-01-24T04:16:39Z

I need to look at the implementation of the block_device configuration but I'd be willing to bet that the ephemerals would also cause this bug if they are exposed as available devices. They should likely be ignored as well. That would make this semi related to #858.

fixes #859 EC2 root block devices are attached automatically at launch [1] and show up in DescribeInstances responses from then on. By skipping these when recording state, Terraform can avoid thinking there should be block device changes when there are none. Note this requires that mitchellh/goamz#214 land first so the proper field is exposed. [1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html

phinze · 2015-01-29T14:22:09Z

Alrighty - this landed yesterday. In 0.3.7 terraform will happily ignore the root volume and get less confused about block devices it should consider changing! 👍

pmoust · 2015-02-16T10:18:04Z

@phinze ouch, we have issues with c88c4a3#diff-5ae59bd3cb446e5b3d5b8521f0b43ac4 since we have defined EBS root block devices in some aws_instances and the output is present in tfstate. When upgrading to current master we get a -/+ due to block_device appearing altered.

resource "aws_instance" "pph_vpn" {
    ami                 = "${var.vpn_ami}"
    instance_type       = "${var.vpn_class}"
    availability_zone   = "${lookup(var.zones, concat("zone", count.index))}"
    subnet_id           = "${element(aws_subnet.pph.*.id, count.index)}"
    associate_public_ip_address = true
    key_name            = "vpn-dec-14" #key to be used to firstly log ing to device with username openvpnas. key is in s3

    user_data           = "${file("../cloud-init/vpn.yml")}"

    tags {
        Name = "vpn0${count.index}"
    }

    block_device {
      device_name = "/dev/sda1"
      volume_type = "standard"
      volume_size = 8
      delete_on_termination = 1
    }
    security_groups     = [ "${aws_security_group.pph_admins.id}",
                            "${aws_security_group.pph_vpn.id}",
                          ]

    count               = 1
}

                "aws_instance.pph_vpn": {
                    "type": "aws_instance",
                    "depends_on": [
                        "aws_subnet.pph",
                        "aws_security_group.pph_admins",
                        "aws_security_group.pph_vpn"
                    ],
                    "primary": {
                        "id": "i-3cf1f8d6",
                        "attributes": {
                            "ami": "ami-1a942472",
                            "associate_public_ip_address": "true",
                            "availability_zone": "us-east-1b",
                            "block_device.#": "1",
                            "block_device.2849382189.delete_on_termination": "false",
                            "block_device.2849382189.device_name": "",
                            "block_device.2849382189.encrypted": "false",
                            "block_device.2849382189.snapshot_id": "snap-5203c08f",
                            "block_device.2849382189.virtual_name": "",
                            "block_device.2849382189.volume_size": "160",
                            "block_device.2849382189.volume_type": "standard",
                            "id": "i-3cf1f8d6",
                            "instance_type": "t2.micro",
                            "key_name": "vpn-dec-14",
                            "private_dns": "ip-10-30-1-53.ec2.internal",
                            "private_ip": "10.30.1.53",
                            "public_dns": "<snipped>",
                            "public_ip": "<snipped>",
                            "security_groups.#": "2",
                            "security_groups.1610004074": "sg-d1b443b5",
                            "security_groups.70589767": "sg-d6b443b2",
                            "subnet_id": "subnet-13b90264",
                            "tags.#": "1",
                            "tags.Name": "vpn00",
                            "tenancy": "default",
                            "user_data": "d012c9fdf3cdde5f3c75743c2329e6879ad5a4cb"
                        }
                    }
                },

plan

-/+ aws_instance.pph_vpn
    ami:                                           "ami-1a942472" => "ami-1a942472"
    associate_public_ip_address:                   "true" => "1"
    availability_zone:                             "us-east-1b" => "us-east-1b"
    block_device.#:                                "1" => "1"
    block_device.2936417914.delete_on_termination: "" => "1" (forces new resource)
    block_device.2936417914.device_name:           "" => "/dev/sda1" (forces new resource)
    block_device.2936417914.encrypted:             "" => "<computed>" (forces new resource)
    block_device.2936417914.snapshot_id:           "" => "<computed>" (forces new resource)
    block_device.2936417914.virtual_name:          "" => "" (forces new resource)
    block_device.2936417914.volume_size:           "" => "8" (forces new resource)
    block_device.2936417914.volume_type:           "" => "standard" (forces new resource)
    instance_type:                                 "t2.micro" => "t2.micro"
    key_name:                                      "vpn-dec-14" => "vpn-dec-14"
    private_dns:                                   "ip-10-30-1-53.ec2.internal" => "<computed>"
    private_ip:                                    "10.30.1.53" => "<computed>"
    public_dns:                                    "<snipped> => "<computed>"
    public_ip:                                     "<snipped>" => "<computed>"
    security_groups.#:                             "2" => "2"
    security_groups.1610004074:                    "sg-d1b443b5" => "sg-d1b443b5"
    security_groups.70589767:                      "sg-d6b443b2" => "sg-d6b443b2"
    subnet_id:                                     "subnet-13b90264" => "subnet-13b90264"
    tags.#:                                        "1" => "1"
    tags.Name:                                     "vpn00" => "vpn00"
    tenancy:                                       "default" => "<computed>"
    user_data:                                     "d012c9fdf3cdde5f3c75743c2329e6879ad5a4cb" => "d012c9fdf3cdde5f3c75743c2329e6879ad5a4cb"

pmoust · 2015-02-16T10:19:21Z

Shall I open a separate issue for this?

AWS provides a single `BlockDeviceMapping` to manage three different kinds of block devices: (a) The root volume (b) Ephemeral storage (c) Additional EBS volumes Each of these types has slightly different semantics [1]. (a) The root volume is defined by the AMI; it can only be customized with `volume_size`, `volume_type`, and `delete_on_termination`. (b) Ephemeral storage is made available based on instance type [2]. It's attached automatically if _no_ block device mappings are specified, and must otherwise be defined with block device mapping entries that contain only DeviceName set to a device like "/dev/sdX" and VirtualName set to "ephemeralN". (c) Additional EBS volumes are controlled by mappings that omit `virtual_name` and can specify `volume_size`, `volume_type`, `delete_on_termination`, `snapshot_id`, and `encryption`. After deciding to ignore root block devices to fix #859, we had users with configurations that were attempting to manage the root block device chime in on #913. Terraform does not have the primitives to be able to properly handle a single collection of resources that is partially managed and partially computed, so our strategy here is to break out logical sub-resources for Terraform and hide the BlockDeviceMapping inside the provider implementation. Now (a) is supported by the `root_block_device` sub-resource, and (b) and (c) are still both merged together under `block_device`, though I have yet to see ephemeral block devices working properly. Looking into possibly separating out `ephemeral_block_device` and `ebs_block_device` sub-resources as well, which seem like the logical next step. We'll wait until the next big release for this, though, since it will break backcompat. [1] http://bit.ly/ec2bdmap [2] http://bit.ly/instancestorebytype Fixes #913 Refs #858

fixes hashicorp#859 EC2 root block devices are attached automatically at launch [1] and show up in DescribeInstances responses from then on. By skipping these when recording state, Terraform can avoid thinking there should be block device changes when there are none. Note this requires that mitchellh/goamz#214 land first so the proper field is exposed. [1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html

AWS provides a single `BlockDeviceMapping` to manage three different kinds of block devices: (a) The root volume (b) Ephemeral storage (c) Additional EBS volumes Each of these types has slightly different semantics [1]. (a) The root volume is defined by the AMI; it can only be customized with `volume_size`, `volume_type`, and `delete_on_termination`. (b) Ephemeral storage is made available based on instance type [2]. It's attached automatically if _no_ block device mappings are specified, and must otherwise be defined with block device mapping entries that contain only DeviceName set to a device like "/dev/sdX" and VirtualName set to "ephemeralN". (c) Additional EBS volumes are controlled by mappings that omit `virtual_name` and can specify `volume_size`, `volume_type`, `delete_on_termination`, `snapshot_id`, and `encryption`. After deciding to ignore root block devices to fix hashicorp#859, we had users with configurations that were attempting to manage the root block device chime in on hashicorp#913. Terraform does not have the primitives to be able to properly handle a single collection of resources that is partially managed and partially computed, so our strategy here is to break out logical sub-resources for Terraform and hide the BlockDeviceMapping inside the provider implementation. Now (a) is supported by the `root_block_device` sub-resource, and (b) and (c) are still both merged together under `block_device`, though I have yet to see ephemeral block devices working properly. Looking into possibly separating out `ephemeral_block_device` and `ebs_block_device` sub-resources as well, which seem like the logical next step. We'll wait until the next big release for this, though, since it will break backcompat. [1] http://bit.ly/ec2bdmap [2] http://bit.ly/instancestorebytype Fixes hashicorp#913 Refs hashicorp#858

ghost · 2020-05-04T02:18:31Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

davedash changed the title ~~Upgrade from 0.3.5->0.3.6 forces recreation of new resources.~~ Upgrade from 0.3.5->0.3.6 forces recreation of new aws_instances with block_storage Jan 23, 2015

mitchellh added the waiting-response An issue/pull request is waiting for a response from the community label Jan 23, 2015

mitchellh added bug provider/aws and removed waiting-response An issue/pull request is waiting for a response from the community labels Jan 23, 2015

ameir mentioned this issue Jan 24, 2015

AWS ephemeral storage #858

Closed

phinze self-assigned this Jan 27, 2015

phinze mentioned this issue Jan 28, 2015

plumb through RootDeviceName on ec2 Instances mitchellh/goamz#214

Merged

phinze mentioned this issue Jan 28, 2015

providers/aws: ignore ec2 root devices #877

Merged

mitchellh closed this as completed in mitchellh/goamz@3d92018 Jan 28, 2015

phinze mentioned this issue Feb 3, 2015

aws_instance block_device declaration causes continual resource re-creation #913

Closed

pmoust mentioned this issue Feb 16, 2015

providers/aws: aws_instance (root) block_device forces new resource on managed infrastructure (no-BC) #978

Closed

phinze mentioned this issue Feb 18, 2015

providers/aws: add root_block_device to aws_instance #998

Merged

ghost locked and limited conversation to collaborators May 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade from 0.3.5->0.3.6 forces recreation of new aws_instances with block_storage #859

Upgrade from 0.3.5->0.3.6 forces recreation of new aws_instances with block_storage #859

davedash commented Jan 23, 2015

davedash commented Jan 23, 2015

davedash commented Jan 23, 2015

mitchellh commented Jan 23, 2015

davedash commented Jan 23, 2015

mitchellh commented Jan 23, 2015

davedash commented Jan 23, 2015

mitchellh commented Jan 23, 2015

patricklucas commented Jan 24, 2015

knuckolls commented Jan 24, 2015

phinze commented Jan 29, 2015

pmoust commented Feb 16, 2015

pmoust commented Feb 16, 2015

ghost commented May 4, 2020

Upgrade from 0.3.5->0.3.6 forces recreation of new aws_instances with block_storage #859

Upgrade from 0.3.5->0.3.6 forces recreation of new aws_instances with block_storage #859

Comments

davedash commented Jan 23, 2015

davedash commented Jan 23, 2015

davedash commented Jan 23, 2015

mitchellh commented Jan 23, 2015

davedash commented Jan 23, 2015

mitchellh commented Jan 23, 2015

davedash commented Jan 23, 2015

mitchellh commented Jan 23, 2015

patricklucas commented Jan 24, 2015

knuckolls commented Jan 24, 2015

phinze commented Jan 29, 2015

pmoust commented Feb 16, 2015

pmoust commented Feb 16, 2015

ghost commented May 4, 2020