aws_elasticsearch_domain keeps reapplying

[thattommyhall]

Similar to #3634

I am using your JSON format to create the aws_elasticsearch_domain

"aws_elasticsearch_domain": {
  "logs": {
    "domain_name": "logs",
    "advanced_options": {
      "rest.action.multi.allow_explicit_index": true
    },
    "access_policies": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Action\":\"es:*\",\"Principal\":\"*\",\"Effect\":\"Allow\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"${aws_eip.public-a-nat.public_ip}\",\"${aws_eip.public-b-nat.public_ip}\",\"${aws_eip.public-c-nat.public_ip}\"]}}}]}",
    "snapshot_options": {
      "automated_snapshot_start_hour": 23
    }
  }
}

It successfully creates, but every time I run it it says it is changing

{
    "Statement": [{
        "Action": "es:*",
        "Condition": {
            "IpAddress": {
                "aws:SourceIp": ["52.48.255.155", "52.48.254.180", "52.48.255.154"]
            }
        },
        "Effect": "Allow",
        "Principal": "*",
        "Resource": "arn:aws:es:eu-west-1:<ID>:domain/logs/*"
    }],
    "Version": "2012-10-17"
}

to

{
    "Statement": [{
        "Action": "es:*",
        "Condition": {
            "IpAddress": {
                "aws:SourceIp": ["52.48.255.155", "52.48.254.180", "52.48.255.154"]
            }
        },
        "Effect": "Allow",
        "Principal": "*"
    }],
    "Version": "2012-10-17"
}

(ie adding the 'Resource')

If I add the Resource entry after it it created then it no longer re-applies, but obviously I can't do it for creation as I dont know the ARN, also if I lookup the ARN, I get Self reference: aws_elasticsearch_domain.logs

This is my first mild snag in ~6 months of using Terraform, it is honestly the best new tech ive used in the last 5y, thanks!

[radeksimko]

I can't do it for creation as I dont know the ARN

I think the easiest way we can get around this would be introducing a new resource, something like aws_elasticsearch_domain_policy which would depend on aws_elasticsearch_domain and would only be responsible to setting the policy.

That way you'd be able to create the domain first and reference the ARN from created domain in the aws_elasticsearch_domain_policy.

It is similar pattern to the one we already use for aws_security_group & aws_security_group_rule. There are some potential issues to be aware of when using this pattern - e.g. we discourage user from setting any rules on the _group level when they use group_rule. We'd be discouraging them from using aws_elasticsearch_domain.access_policy if they want to use (hypothetical) aws_elasticsearch_domain_policy here as well.

---

Longer term I think this will need solving in more wider context, it is also causing similar issues in most IAM policies where user needs to do a lot of self-referencing. See #3267

[tj]

👍 unusable right now as-is, it wouldn't be a big deal but elasticsearch takes a good 15-20 minutes to apply these changes haha :D

[gposton]

Moving some comments over from #3539. Hope they're useful.

I'm seeing the same issue.... Looking at the output from the terraform apply, the access_policy is updated despite the fact that it is logically equivalent (no update should be required).

The order of the elements in the access policy json are different. Also, it looks like the Resource ARN is being automatically injected into the access_policy. That explains why terraform see's the diff and triggers the modify. It would probably work to add the resource arn to the policy in the tf template after the initial creation of the ES domain (can't add it at the initial creation because you don't know what it is until after the resource is created).

I can reproduce this on each terraform apply.

Here is the output from the terraform apply:

[ats_dependencies] access_policies: "{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"","Action":"es:","Resource":"arn:aws:es:us-west-2:763429161784:domain/ats-prod/","Condition":{"IpAddress":{"aws:SourceIp":["52.36.237.75/32","52.32.171.250/32","52.34.208.134/32","50.112.131.3/32","54.201.98.238/32","54.69.54.64/32","52.24.90.172/32"]}}}]}" => "{"Statement":[{"Action":"es:","Condition":{"IpAddress":{"aws:SourceIp":["52.36.237.75/32","52.32.171.250/32","52.34.208.134/32","50.112.131.3/32","54.201.98.238/32","54.69.54.64/32","52.24.90.172/32"]}},"Effect":"Allow","Principal":"*"}],"Version":"2012-10-17"}"

[dvalentin]

Same issue here:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "${aws_iam_role.ecs_instance.arn}", "AWS": "${aws_iam_role.kinesis2elasticsearch.arn}" }, "Action": "es:*", "Resource": "*" }, { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": "es:*", "Condition": { "IpAddress": { "aws:SourceIp": [ "${var.allowed_cidr.office}" ] } }, "Resource": "*" } ] }
This policy will be applied on every run (the ressources referenced in that policy, don't change that frequently) and recreates the cluster (not terraforms fault!).
Making it an own terraform ressource might be a possible fix.

[gposton]

Workaround is to put the ARN in the access policy.

This can be done as the ARN follows a known convention.

i.e.

access_policies = <<CONFIG
{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Principal": "*",
    "Action": "es:*",
    "Resource": "arn:aws:es:${var.region}:${var.aws_account_id}:domain/${var.cluster_name}-${var.environment}/*",
    "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "${element(formatlist("%s/32", split(",", terraform_remote_state.nat.output.nat_public_ips)), 0)}",
            "${element(formatlist("%s/32", split(",", terraform_remote_state.nat.output.nat_public_ips)), 1)}",
            "${element(formatlist("%s/32", split(",", terraform_remote_state.nat.output.nat_public_ips)), 2)}",
            "${join("\",\"", formatlist("%s/32", split(",", var.source_ips)))}"
          ]
        }
      }
    }
  ]
}

[rottenbytes]

👍 on introducing a aws_elasticsearch_domain_policy resource type

[cahlbin]

+1 on introducing aws_elasticsearch_domain_policy

[cahlbin]

I tried the workaround @gposton suggested, to inline the policy and only use predefined vars, but it seems as if terraform still re-applies the access policy on each terraform apply, despite the ES cluster already existing since a previous apply and no change in inline policy

[gposton]

@cahlbin, take a look at the policy diff during terraform plan, and make sure the ordering of the keys in the access policy in the terraform template match what terraform is seeing in AWS. I'm not sure if the ordering is important or not, but I have a hunch that it might be.

[daveadams]

My impression from running into this in the past is that the AWS API is returning the JSON document in different equivalent orderings each time, (ie with the keys in a different order). If I run terraform plan a dozen times, I might be told nothing needs to change two of those twelve times, and that a change needs to happen the other ten times. The more complex the policy document, the less often terraform plan will appear to agree with the existing resource.

I don't think adding an aws_elasticsearch_domain_policy resource will help in this case, because applying the policy appears to be the operation that takes 10+ minutes to complete, and that will not change regardless of how the Terraform-side logic is structured.

Instead, I think the resource needs to stop treating the JSON as a meaningless string and applying a change when the AWS API returns a permuted variation of the original, and needs to assume AWS treats equivalent JSON documents equivalently and parse the JSON and compare the resulting data structures for equivalence.

[n8gard]

I'm having this problem, too. Here is my code. This is a real bummer. I'm going to have to abandon using the ElasticSearch service and build ElasticSearch servers by hand. This makes me very, very sad at Terraform. I can't have 10 minute (or longer) apply operations.

resource "aws_elasticsearch_domain" "es-elk" {
    domain_name = "es-elk-${var.environment}"
    cluster_config = {
      instance_type = "${var.es_data_node_instance_type}"
      instance_count = "${var.es_data_node_instance_count}"
      dedicated_master_enabled = true
      dedicated_master_type = "${var.es_master_node_instance_type}"
      dedicated_master_count = "${var.es_master_node_instance_count}"
      zone_awareness_enabled = true
    }

    ebs_options = {
      ebs_enabled = true
      volume_type = "gp2"
      volume_size = "${var.es_volume_size}"
    }

    advanced_options {
        "rest.action.multi.allow_explicit_index" = true
    }

    access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {
                  "aws:SourceIp": [
                    "${var.cidr_block_elk_one}",
                    "${var.cidr_block_elk_two}",
                    "${var.cidr_block_elk_three}"
                  ]
                }
            }
        }
    ]
}
CONFIG
    snapshot_options {
        automated_snapshot_start_hour = 23
    }
    tags {
      Domain = "${var.environment}"
    }
}
output "es-elk-endpoint" {
    value = "${aws_elasticsearch_domain.es-elk.endpoint}"
}

[briankmc]

@n8gard this is fixable for me by being very careful with the ordering and formatting of access_policies.

This gets me a non-cycling terraform plan with v0.6.16:

resource "aws_elasticsearch_domain" "logs" {
  domain_name = "${var.name}-logs" # must match Resource in access_policies

  access_policies = <<EOF
{
  "Statement": [
    {
      "Action": "es:*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "127.0.0.1/32"
        }
      },
      "Effect": "Allow",
      "Principal": "*",
      "Resource": "arn:aws:es:${var.region}:${var.aws_account_number}:domain/${var.name}-logs/*"
    }
  ],
  "Version": "2012-10-17"
}
EOF

  [...]
}

[n8gard]

@briankmc that worked! Would like to understand why but I'm happy to leave it as esoteric weirdness. Thanks!

[devinsba]

Even being very careful as @briankmc suggests. If I don't add the ARN it will apply every time.

On the other hand I've added it but now run into aws_elasticsearch_domain.es: diffs didn't match during apply. for some reason.

I have a list with 2 statements. This only happens if I have the ARN on the first one. Essentially

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "${role_arn}"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:${region}:${account_id}:domain/${domain_name}/*"  <--- IF I ADD THIS IT FAILS
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:${region}:${account_id}:domain/${domain_name}/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": ${ip_json_list}
        }
      }
    }
  ]
}

It will modify every time if it's only on the second one. This has been a huge headache for me this week. Is #8648 still the PR that should get this done?

EDIT: It appears that this had something to do with my use of a rendered template, if instead I just inline the policy it works fine. weird.

[daveadams]

Is that your actual code? I believe you need to prefix the var names with
"var." as in:

"Resource":

"arn:aws:es:${var.region}:${var.account_id}:domain/${var.domain_name}/*"

On Thu, Oct 13, 2016 at 1:41 PM, Brian Devins notifications@github.com
wrote:

Even being very careful as @briankmc https://github.com/briankmc
suggests. If I don't add the ARN it will apply every time.

On the other hand I've added it but now run into
aws_elasticsearch_domain.es: diffs didn't match during apply. for some
reason.

I have a list with 2 statements. This only happens if I have the ARN on
the first one. Essentially

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "${role_arn}"
},
"Action": "es:",
"Resource": "arn:aws:es:${region}:${account_id}:domain/${domain_name}/" <--- IF I ADD THIS IT FAILS
},
{
"Effect": "Allow",
"Principal": {
"AWS": ""
},
"Action": "es:",
"Resource": "arn:aws:es:${region}:${account_id}:domain/${domain_name}/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": ${ip_json_list}
}
}
}
]
}

It will modify every time if it's only on the second one. This has been a
huge headache for me this week. Is #8648
#8648 still the PR that
should get this done?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#5067 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAETooiJZCpSBvOeSTU1RkC2TVwyTPY2ks5qzntXgaJpZM4HWVvH
.

[devinsba]

@daveadams Seems my edit wasn't fast enough to make it in the email. This was doing a rendered template. the workaround works once I remove that abstraction.

[orlando]

in addition to the @briankmc workaround, there's a data resource called aws_caller_identity which exposes the account_id. so you can write your resources like

data "aws_caller_identity" "current" {}

resource "aws_elasticsearch_domain" "es" {
  domain_name = "${var.domain_name}"
  ...
  access_policies = <<CONFIG
    {
      "Statement": [
        {
          "Action": "es:*",
          "Condition": {
            "IpAddress": {
              "aws:SourceIp": ["127.0.0.1/32"]
            }
          },
          "Effect": "Allow",
          "Principal": "*",
          "Resource": "arn:aws:es:ap-southeast-1:${data.aws_caller_identity.current.account_id}:domain/${var.domain_name}/*"
        }
      ],
      "Version": "2012-10-17"
    }
  CONFIG
  ...
}

So, you don't need to provide extra variables to get this done

[mitchellh]

It appears this is fixed by using data sources. If someone can get me a newer version diff's didn't match error then I can take a look but I have a strong suspicion that this is resolved in newer versions.

[orlando]

The problem is that the resource gets reapplied, unless you specify the "Resource" key in your access policy. It is usable with the workaround, but not sure if that's the expected behaviour

[mitchellh]

@orlando Ah, I missed that, reopening.

[ghost]

I know the issue has been closed, but we are facing the same issue unfortunately. We are using aws_elasticsearch_domain_policy together with aws_iam_policy_document. I don't like inline JSON to construct policies, so we are using a data document

Data block

data "aws_iam_policy_document" "ElasticSearchPolicy" {

  statement {
    effect = "Allow"

    actions = [
      "es:ESHttpPost",
      "es:ESHttpPut"
    ]

    principals {
      identifiers = [
        "${var.lambda_fcn_arn}"
      ]

      type = "AWS"
    }

    resources = [
      "${aws_elasticsearch_domain.es.arn}/*"
    ]
  }

  statement {
    effect = "Allow"

    actions = [
      "es:*"
    ]

    condition {
      test = "IpAddress"
      values = [
        "${var.list_one}",
        "${var.list_two}"
      ]

      variable = "aws:SourceIp"
    }
  }

}

Domain Policy

resource "aws_elasticsearch_domain_policy" "eslogextract" {
  domain_name = "${aws_elasticsearch_domain.es.domain_name}"
  access_policies = "${data.aws_iam_policy_document.ElasticSearchPolicy.json}"
}

Every time we run a Terraform apply, it recreates and reapplies the policy to the ElasticSearch domain. This takes at least 10 minutes every time we run a terraform apply

[willemveerman]

I think this ticket has been closed prematurely.

The issue raised by the OP still occurs.

It seems that this ticket was closed because the Hashibot created a corresponding ticket in the terraform-provider-aws section. However, that ticket refers to a different issue: it's concerned with the policy change taking a long time.

Moreover, in another comment elsewhere one of the Terraform team refers a user to this ticket for updates regarding the ES policy-recycling issue.

Therefore, this ticket should be re-opened if this issue is not fixed and not covered by any other ticket.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.