Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delayed execution of reading for file variables #3354

Closed
lusis opened this issue Sep 29, 2015 · 12 comments
Closed

Delayed execution of reading for file variables #3354

lusis opened this issue Sep 29, 2015 · 12 comments

Comments

@lusis
Copy link
Contributor

lusis commented Sep 29, 2015

In an attempt to build some really reusable modules, we've broken a few things out. One thing we need is the ability to generate an AWS keypair for each environment we stand up. I discovered the null resource and thought it would help by wrapping the ssh-keygen process up into a resource that could then be used for aws_key_pair. Evidently this doesn't appear to work as ${file("foo")} contents are required to be read before work can begin:

resource "null_resource" "generate_keypair" {
    provisioner "local-exec" {
        command = "ssh-keygen -b 4096 -f ${path.cwd}/${var.spath_orgname}-deployer-key"
    }
}

resource "aws_key_pair" "deployer" {
  key_name = "${var.spath_orgname}-deployer-key"
  public_key = "${file("${path.cwd}/${var.spath_orgname}-deployer-key.pub")}"
  depends_on = ["null_resource.generate_keypair",]
}

terraform plan and terraform apply both fail as the file does not yet exist (apparently though I've not dug into the code yet) for reading. This is a normally safe sanity check so I totally understand WHY.

Is there an option for lazy evaluation or is there a workaround that involves a dance around generating the keypair files locally and then some how cat'ing them out into a variable?

@apparentlymart
Copy link
Contributor

A possible hack around this would be to use the template_file resource with depends_on your null_resource, treating the file as a template that happens to have no interpolations in it.

Having a resource for reading files is probably the only way to make this delayed read work within the current config mechanism, since functions can't express dependencies.

@phinze
Copy link
Contributor

phinze commented Oct 12, 2015

Yep @apparentlymart's suggestion is how I'd do it - use template_file to get your file treated as a node on the dependency graph.

@phinze phinze closed this as completed Oct 12, 2015
@afiune
Copy link

afiune commented Feb 4, 2016

@lusis @apparentlymart @phinze Hey guys... I happened to find this topic very useful. I tried the suggestion myself but I might be doing something wrong. 😄

Having this code:

# Generate key
resource "null_resource" "generate_key" {
  provisioner "local-exec" {
    command = "ssh-keygen -t rsa -N '' -b 2048 -f my_key -y"
  }
}

# Template to delay action of reading the generated key
resource "template_file" "my_key" {
  depends_on = ["null_resource.generate_key"]
  template = "${my_key}"
  vars {
    my_key = "${file("my_key")}"
  }
}

I would imagine from here I can just use "${template_file.my_key.rendered} but I am still having an error trying to run plan:

$ terraform plan
There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * file: open my_key: no such file or directory in:

${file("my_key")}

Am I doing something wrong? (using Terraform version v0.6.9)

@HX-Rd
Copy link

HX-Rd commented Apr 28, 2016

@afiune Yeah having the same "no such file or directory in" problem. Looks like there is a check done on the paths in the variable interpolation in a phase before the actual run. If you create a empty "my_key" before the terraform apply, that key will be overwritten and used. Ugly hack but the only one that I have come up with to solve this.

@afiune
Copy link

afiune commented Apr 28, 2016

@HX-Rd It is exactly what I did. But it is very ugly indeed. Thanks for the answer though! 👍

@rneu31
Copy link

rneu31 commented Nov 14, 2016

@apparentlymart @afiune @HX-Rd @phinze Does anyone know if this "hack" is still functional in the newest Terraform? I'm having quite the time trying to do something like this.

I am trying to have an md5 generated for a directory each time terraform runs, save the hash to a file, and use that file's contents to determine if provisioning needs to happen. If this is a far off use case feel free to let me know.

My problem seems to be the policy-hash-file gets refreshed first. If I run twice in a row, the second time will pick up the necessary changes. Verified this by inspecting the state file directly.

resource "null_resource" "hash-policy" {
  provisioner "local-exec" {
    command = "find ../../../../policy -type f -exec md5sum {} ; | sort -k 2 | md5sum > policy-hash.txt"
  }

  triggers {
    uuid = "${uuid()}" # trigger always
  }
}

data "template_file" "policy-hash-file" {
  depends_on = ["null_resource.hash-policy"]
  template = "${file("${path.root}/policy-hash.txt")}"
}

resource "null_resource" "run-chef-client" {   
  provisioner "remote-exec" {
    # run chef client
  }

  triggers {
    random_file =  "${data.template_file.policy-hash-file.rendered}"
  }
}

This is all an elaborate scheme to run Chef when contents of a specific folder changes.

Thanks.

@apparentlymart
Copy link
Contributor

In order for the hack to work it is necessary to use the older (now deprecated) form of template_file that takes a filename directly as an argument, rather than using the file function.

In the future I'd like to be able to address this sort of problem using the mechanism added by #8768.

@irvingpop
Copy link

irvingpop commented Jul 4, 2017

So I got this working in Terraform 0.9.10 with some slight tweaks on the above technique. Namely by using a combination of a computed resource for the filename and by depending on an intermediary resource for data.

It looks like this:

resource "null_resource" "generate_chef_keypair" {
  provisioner "local-exec" {
    command = "ssh-keygen -t rsa -N '' -f .chef/delivery-validator-${random_id.automate_instance_id.hex}.pem"
  }
}

resource "aws_instance" "chef_server" {
  # resources to use the pre-created key
  depends_on = ["null_resource.generate_chef_keypair"]
}

# template to delay reading of validator key
data "template_file" "delivery_validator" {
  template = "${file(".chef/delivery-validator-${random_id.automate_instance_id.hex}.pem")}"

  vars {
    hacky_thing_to_delay_evaluation = "${aws_instance.chef_server.private_ip}"
  }
  depends_on = ["aws_instance.chef_server"]
}

# other resources that use ${template_file.delivery_validator.rendered}

of course this will surely break in 0.10.x :)

@irvingpop
Copy link

Fellow travelers who land here, an update.

As expected the hacky thing above (left for posterity) did in fact break with 0.10.x, but the External Data Source provider is awesome (once you get the hang of it).

I've built an updated example here: https://gist.github.com/irvingpop/968464132ded25a206ced835d50afa6b

@dustydecapod
Copy link

Just wanted to drop in and mention that I've accomplished this using the TLS provider.

Relevant docs:
https://www.terraform.io/docs/providers/tls/
https://www.terraform.io/docs/providers/tls/r/private_key.html#

@mukund1989
Copy link

@ZoidBB That provider stores the key in statefile

@ghost
Copy link

ghost commented Apr 2, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

9 participants