Rich template language

[hashibot]

This issue was originally opened by @errordeveloper as hashicorp/terraform#1845. It was migrated here as part of the provider split. The original body of the issue is below.

---

As #1778 has provided a way of doing interpolation in files, next step would be to see whether there could be a richer language available from inside a template file.

Such language would help for generating provisioning data that could be passed to a provide as metadata for setting up VMs or basic configuration files to be uploaded to VMs.

[hashibot]

This comment was originally opened by @errordeveloper as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

In one instance I have implemented a shell script that looks rather quite ugly for what it's able to do. It is being called from infra/main.tf#L44 and main.tf#L126.

[hashibot]

This comment was originally opened by @errordeveloper as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I should also clarify that in my use-case it's CoreOS that is being provisioned, and many other folks would probably suggest that a configuration management tool is what I want. However, I believe that this this kind of mechanism would be of use when one needs to seed the initial configuration files for their configuration management tool to run.

[hashibot]

This comment was originally opened by @apparentlymart as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

It could be nice to use the same template language as consul-template, but with access to Terraform resources and variables within the calling module rather than access to Consul concepts like services and keys.

This could then allow Terraform provisioning to do the same sorts of things consul-template does, but for configurations that only vary per-deploy, rather than those that can vary at runtime. Having the template be able to access the resources directly means being able to easily iterate over list attributes and resource wildcards, which wouldn't be so easily to achieve with the explicit variables map in the existing template implementation.

Example haproxy template fragment, based on the one for consul-template:

listen http-in
    bind *:{{var "load_balancer_listen_port"}}{{range resource "aws_instance.server.*"}}
    server {{.id}} {{.private_ip}}:{{var "server_instance_listen_port"}}{{end}}

Of course, to achieve this would require a rather unusual resource type whose dependencies are defined by the contents of the external template file, rather than by interpolations within the resource definition itself.

[hashibot]

This comment was originally opened by @mitchellh as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I think we should just extend the template_file resource to support a type parameter that specifies the template type and we can support a variety there.

[hashibot]

This comment was originally opened by @errordeveloper as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Yes, 👍, as long as there is a way of saying that "here is a file that needs to be processed in some way before it can be used". May be the types will have to declared somewhere, so user sees "You need tool foo to run this", instead of "foo: command not found" or something totally weird.

[hashibot]

This comment was originally opened by @progrium as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

template_file with a way to provide types would be nice, but ideally if shell were one of them. Or perhaps support https://github.com/gliderlabs/sigil directly? We could probably do either or both as a PR depending on what sounds right.

[hashibot]

This comment was originally opened by @7heo as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@mitchellh 👍 that would be really great.

[hashibot]

This comment was originally opened by @ketzacoatl as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Saltstack makes the template/rendering engine plugable, so users can pick which renderer to use - it has worked out very well in practice and I believe that would be a great addition to Terraform's template provider!

[hashibot]

This comment was originally opened by @lbernail as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

A richer templating engine would definitetly help!
👍

[hashibot]

This comment was originally opened by @errordeveloper as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

We had been using Jsonnet in Kubernetes Anywhere project, and it provides quite a few extra features for generating Terraform JSON then HCL does, it also have richer tooling for inlining plain text files, escaping JSON and rendering templates etc.

If you create a file named hello.txt with the following content:

Hello, my name is %(name)s, I am %(age)d years old.

you can import it from hello.jsonnet using importstr and substitute age and name variables from a dictionary to create an object with a meaningful message:

{
  message: (importstr "hello.txt") % {age: 25, name: "Rose"}
}

Run jsonnet hello.jsonnet and you will get:

> jsonnet test.jsonnet
{
   "message": "Hello, my name is Rose, I am 25 years old.\n"
}

There other nice methods like std.escapeStringDollars and std.escapeStringBash.

[hashibot]

This comment was originally opened by @errordeveloper as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Please note, that with Jsonnet you would still make some use of Terraform interpolation, but only in case where you have to enforce dependencies. I think this is something worse documenting a bit more. Unlike some other kind of methods one could find for generating Terraform code, Jsonnet still keeps it declarative.

[hashibot]

This comment was originally opened by @flypenguin as hashicorp/terraform#1845 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I'd also go for @apparentlymart's suggestion to use the consul templating languate. Make Hashi products consistent, and while it might not be the most intuitive one, it's "part of the family", already out there and being used.

[noelmcloughlin]

I support suggestion from @ketzacoatl to "make the template/rendering engine plugable" in terraform. This means designing for provider plugin abstraction within TF.

@apparentlymart suggests using Consul templating engine. I personally use jinja2 with saltstack (and nunjunks with expressJS). The suggestion of a plugable provider interface facilitates everyone who has opinion on the topic, and allows users to bring their templating preference to TF.

[apparentlymart]

Hi all,

Since this provider uses the syntax from Terraform's existing expression interpolator, it will get more features in the release of this provider that accompanies the Terraform Core 0.12 release, including what's described in our preview article.

This doesn't mean that we can't/won't also support other template engines here, but at least the current built-in one will have support for conditionals and iteration, and will thus meet most of the use-cases we've seen.

However, on the topic of making the template engine pluggable, I expect if we were to go down that road we'd probably just say that each template engine ought to have its own Terraform provider, since that's a plugin mechanism that Terraform already supports and it seems unnecessarily confusing to have this template provider also have its own separate level of plugins, which we'd then need to have a mechanism to find and install, etc. template_file would then be specifically for Terraform's template engine, with other engines each having their own providers.

It's already possible to write a Terraform plugin for any template engine you like as long as there is a Go implementation available for it. A challenge is that Terraform passes data to plugins in terms of its own type system (rather than Go's type system) and so it'll probably require some marshaling effort to prepare the incoming data for what another Go-based template system is expecting. The template provider avoids this because it's using Terraform's own language, and thus it already natively supports all of the types Terraform does in the representation Terraform uses, but such conversion is technically possible and mainly consists of just figuring out what is the most intuitive mapping for good usability.

For example, someone could in principle write a pongo2 plugin for Terraform which offers a pongo2_template data source that renders templates using pongo2:

# not yet implemented
data "pongo2_template" "example" {
  template = "${file("${path.module}/example.tmpl")}"
  vars = {
    foo = "bar"
  }
}

Once hashicorp/terraform#15252 is implemented (design effort for which is getting underway), it would even be possible for this to be done entirely by a third-party, though that doesn't mean we wouldn't consider bringing it into the terraform-providers set if there were enough demand once it is written.

[ketzacoatl]

Yes, and yes, that all sounds reasonble and great. Very excited for the changes coming in 0.12 and seeing how that opens some of these doors.