Proposal: "external" provider

[apparentlymart]

We repeatedly see people trying to use external code (usually via local-exec provisioners, but sometimes via wrapper scripts) to extend Terraform for their use-cases in a lighter way than writing provider code in Go.

This is a proposal for a more "official" way to do these lightweight integrations, such that they can work within the primary Terraform workflow.

The crux of this proposal is to define a "gateway interface" between Terraform and external programs, so that a user can a write separate program in any language of their choice that implements the gateway protocol, and have Terraform execute it. This approach is inspired by designs such as inetd and CGI which interact with a child process using simple protocol primitives: environment variables, command line arguments, and stdio streams.

external_data_source data source

data "external_data_source" "example" {
  program     = "${path.module}/example.py"
  interpreter = "python"

  query {
    # Mapping of values that are of significance only to the external program
    foo = "baz"
  }
}

When evaluating this data source, Terraform will create a child process and exec python ./example.py, writing a JSON-encoded version of the contents of query to the program's stdin.

The program should read the JSON payload from stdin, do any work it wants to do, and then print a single valid JSON object to stdout before exiting with a status of zero to indicate success.

Terraform then parses that JSON payload and exposes it in a map attribute result, from which the results can be interpolated elsewhere in the configuration:

    bar = "${external_data_source.example.result.bar}"

It's the responsibility of the developer of the external program to make sure it acts in a side-effect-free fashion, as is expected for data source reads.

external_resource resource

resource "external_resource" "example" {
  program     = "${path.module}/example.py"
  interpreter = "python"

  arguments {
    # Mapping of values that are of significance only to the external program
    foo = "baz"
  }
}

When evaluating this resource, there is a separate protocol for each of the resource lifecycle actions. The common aspect of all of these is that Terraform creates a child process and runs python ./example.py with a single additional argument containing the name of the action: "create", "read", "update" or "delete".

Just as with the data source above, there is a map attribute result. For resources, the result attribute corresponds to Computed attributes in first-class resources, while arguments corresponds to Optional (though the external program may have some additional validation rules it enforces once it's run).

The protocol for each of these actions described in the following sections.

read

For read, the given program is run with read as an argument, and Terraform writes to its stdin a JSON payload like the following, using arguments and id from the existing state:

{
    "id": "abc123",
    "arguments": {
        "foo": "baz"
    }
}

The program must print a valid JSON mapping to its stdout, with top-level keys id, arguments and result, as follows:

{
    "id": "abc123",
    "arguments": {
        "foo": "baz"
    },
    "result": {
        "bar": "xyxy"
    }
}

The program must exit with a status of zero to signal success in order for Terraform to accept the result.

Terraform updates the arguments and result mappings in the state to match what's returned by the program here.

It's the responsibility of the developer of the external program to make sure that read acts in a side-effect-free fashion.

create and update

Create and update follow a very similar protocol: the given program is run with either create or update as a command line argument, and Terraform writes to its stdin a JSON payload like the following:

{
    "id": "abc123",
    "arguments": {
        "foo": "baz"
    },
    "old_arguments": {
        "foo": "baz"
    }
}

In both cases the "arguments" come from the configuration. In the "update" case the "id" and "old_arguments" come from the state, while in the "create" case they are omitted altogether.

After doing whatever side-effects are required to effect the requested change, the program must respond in the same way as for the read action, and Terraform will make the same changes to the state.

delete

Delete is the same as update except that the command line argument is delete and the program is not expected to produce any output on stdout.

If the exit status is zero then Terraform will remove the resource from the state.

Child Process Environment

For both the data source and the resource, the child process inherits the environment variables from the Terraform process, possibly giving it access to e.g. AWS security credentials.

The current working directory is undefined, and so gateway programs shouldn't depend on it. As with most cases in Terraform, it'd be better to pass the result of the file(...) function into the program as an argument rather than have it read files from disk itself, though the program may also choose to build paths relative to its own directory in order to load resources, etc.

Error Handling

If the target program exits with a non-zero status, Terraform collects anything written to stderr and uses it as an error message for failing the operation in question.

For operations where a valid JSON object is expected on stdout, any parse error is also surfaced as an error.

Example Python Data Source

import json
import os

arguments = json.load(os.stdin)
foo = arguments["foo"]

# .. do something with "foo" ..

json.dump({
    "bar": "xyxy",
}, os.stdout)

General Thoughts/Observations

Intended Uses

There are two high-level use-cases for this sort of feature, based on examples I've seen people share elsewhere in the community:

• Terraform supports the service I want to use but doesn't support some detail of it that I care about.
• I want to integrate Terraform with an internal, proprietary system.

For the former case, it would be good if people would still share their use-cases as feature requests in this issue tracker so that we can eventually implement all of the capabilities of the services we support. In this case I hope the user would aspire to eliminate the use of this provider eventually.

The latter case seems like the main legitimate reason to use the "external" provider as a "final solution", particularly if an organization already has tooling in place written in another language and doesn't have the desire or resources to port it to Go.

Effect on the Terraform Ecosystem?

Were this provider to be implemented, it could be used as a crutch for integrating with systems where a Terraform provider is not yet available. Pessimistically, this could cause a lower incentive to implement "first-class" Terraform providers, hurting the Terraform ecosystem in the long run.

However, my expectation is that this interface is clumsy and inconvenient enough that it will be tolerated for short-term solutions to small problems but that there will still be a drive to implement first-class provider support for complex and common services.

Higher-level Abstractions

Just as happened with CGI, it's possible that motivated authors may create higher-level abstractions around this low-level gateway protocol in their favorite implementation language. For example, a small Python library could allow a resource to be implemented as a Python class, automatically dealing with the reading/writing and JSON serialization behind the scenes.

With that said, this protocol is intended to be simple enough to implement without much overhead in most languages, so I expect most people wouldn't bother with this sort of thing and would just code directly to the protocol.

One area that could be interesting is something that can take the arguments and old_arguments properties of the update request payload and make an interface like Terraform's schema.ResourceData, though I'd hope that people would consider writing a real Terraform provider if they find themselves doing something that complicated.

---

Relevant other issues:

• No way to use data "template_file" on a file generated on the fly by local-exec as it fails during planning #8406 use an external program to generate a template for template_file

[mrwilby]

+1 to this bad boy. Would really help obviate the need for complex workarounds.

[sl1pm4t]

@apparentlymart - looks good, and reminds me a lot of how drone.io implemented their plugin system.
A couple of comments:

• In your external_resource example config above, the config block is a terraform data block, rather than resource. Is that what you intended?
• With a Read operation on the external resource, there needs to be a way to signal back to terraform that the real resource has gone away. i.e. could the external provider return an empty ID value, as is common with the internal resource providers?

[radeksimko]

@apparentlymart I understand where you're coming from, it sounds like #1156 I raised a while ago which eventually emerged into getting null_resource documented.

I have mixed feelings when thinking about it ~1year after opening the mentioned issue. Part of me agrees with @phinze

I'm still sort of concerned that implementing either one of these features could end up allowing people to shoot themselves in the foot too much.

and the other part of me agrees with you - #1156 (comment)

---

To comment on the actual proposal:

I would be concerned about breaking cross-platform compatibility + dependency expectations. This is one of Terraform's/Go advantages people don't always realise and they may get bitten by when they try to implement their own Terraform 😃 . e.g. In your example above you just expect the system to provide Python of a particular version, which may grow into expecting some PyPI dependencies too. The same story applies to pretty much any interpreted language (shell included).

[apparentlymart]

@radeksimko yeah, the dependency on python and whatever else is something that would make this hard to use in my company's production environment, where we run Terraform on deliberately-very-minimal dedicated machines.

My assumption here was that people were using local-exec to do these things anyway, so whoever is doing that must already be running Terraform in such a way that external scripts will work. My feeling was that this is just part of why using this resource would not be a magic bullet and that users would have to make some careful tradeoffs in using it (which could be discussed in detail in the docs), one of which would be ensuring that Terraform only runs in appropriately-configured contexts.

(I would hope that in most "production-ish" situations Terraform isn't being running on assorted different machines anyway, and so it shouldn't be incredibly hard to ensure that it has an appropriate environment around it for something like this. That's already true if e.g. you have out-of-tree Terraform plugins that your configuration depends on.)

I think the key difference in this proposal over what you proposed in #1156 is that in your proposal you were essentially just providing an alternative syntax for running arbitrary commands, whereas the protocol described here makes the full data source and resource lifecycles available to external glue code, thus hopefully making these external scripts act as more well-behaved elements of Terraform's model... hopefully resulting in less self-foot-shooting. 😀

Continuing the theme of self-foot-shooting: I suppose having seen loads of examples of what people are doing to work around this problem, I'm feeling like Terraform should provide a recommended path that can come with suitable docs that explain the caveats. That way at least people are going into this with eyes wide open, and not instead getting something to work and then discovering only later the implications of what they built.

[mtougeron]

FWIW, this would be something that I could use to solve me Chef de-provisioning issues from #4121 (comment) after a stack has been destroyed.

[StyleT]

@apparentlymart I suggest instead of supporting multiple interpreters do the following:

• drop interpreter parameter
• replace program with command like in the local-exec

This will allow to use any program you want which supports I/O standard you described in 1st post. And formally TF still will be dependencies free :)

[StyleT]

Also maybe it's reasonable to start with data provider only and implement resource in 2nd development iteration. Looks like external_resource requires much more discussion/development because of complexity. On the other hand data source looks very simple and can be easily implemented.

[apparentlymart]

@StyleT the data source is indeed simpler. I'd like to discuss them both together for a little while because the two should ideally end up having a very similar protocol, but you are right that there's no need for them to be implemented at the same time.

[mfischer-zd]

This is a fantastic proposal, and I look forward to its implementation. 👍

There are, of course, alternate ways to accomplish the goal of feeding inputs to Terraform, such as wrapping Terraform with programs that populate TF_VAR_* environment variables; but in my experience, different teams may prefer different approaches such as this one.

I don't share other commenters' concern about whatever other dependencies this proposal might allow others to introduce into their environment. To borrow from the Python example, if someone wants to build a stack that uses Terraform and Python together, that's their own business, and it would be unnecessarily prescriptive of us to prevent them from doing so. (Not that we could, anyway, because Terraform might be wrapped by a Python script.)

Forcing dependencies on a user, of course, is another matter entirely; but this proposal doesn't seek to do that.

[toddnni]

I noticed that we have new external provider in 0.8.0, great!

I have created a shell provider https://github.com/toddnni/terraform-provider-shell that can be used to wrap external programs to terraform. I would be happy to have same functionality covered in external provider either by taking the functionality from mine or having better in the existing provider.

By the way, there is even similar arguments {} structure in the shell provider, but it doesn't support update operation at all (and is not even designed to because maybe in that case it is better to have a native provider).

[apparentlymart]

Thanks for that, @toddnni!

The work for 0.8 just added the data source proposed here. I am planning to add support for a resource too, but wanted to see how the data source plays out first in case there are some design improvements we can learn from experience of its use.

Next time I'm looking at this I'll have a look at your provider in some more detail. It looks like in your case you went for generic shell-style wrapping and just capturing the stdout verbatim, whereas my proposal here has a more structure protocol that requires a more specialized external program. I think both are reasonable approaches with different tradeoffs... I'm curious to see how people will use the external provider and whether the additional structure is warranted vs. just capturing stdout as you did.

[mtougeron]

@apparentlymart If it helps, here's how I'm using this data source. It isn't perfect (or really even a best practice) but this let's us automate the installation of the pip modules for an aws lambda function and trigger an upload if one was done.

pip.tf

data "external" "pip-install" {
  program = ["python", "${path.module}/pip-install.py"]

  query = {
    source_dir = "${path.module}/../src/"
  }
}

resource "random_id" "zipfile-id" {
  keepers = {
    version = "${md5(file("${path.module}/../src/${var.module-name}.py"))}"
    pip_installed = "${data.external.pip-install.result.installed_package}"
  }

  byte_length = 8
}

pip-install.py

#!/usr/bin/env python
import sys
import os
import json
import pip

from pprint import pprint

def _pip_install(package, dest_dir):
    pip.main(['install', package, '--target', dest_dir, '--quiet'])

def _get_stdin_params():
    if sys.stdin.isatty():
        return {}

    lines = [x.strip() for x in sys.stdin.readlines() ]

    lines = filter(None, lines)
    if len(lines) == 0:
        return {}
    else:
        return json.loads(','.join(lines))

def main():
    params = _get_stdin_params()
    if 'source_dir' in params:
        source_dir = os.path.realpath(params['source_dir'])
    else:
        source_dir = os.path.dirname(os.path.realpath(__file__))

    filename = '{}/requirements.txt'.format(source_dir)
    if not os.path.exists(filename):
        return False

    with open (filename, 'r') as file:
        reqs = [x.strip() for x in file.readlines()]
        reqs = filter(None, reqs)

    installed_package = False
    for req in reqs:
        if '=' in req:
            parts = req.split('=', 1)
            # Take just the first part without the version for the dir name
            req_dir = "{}/{}".format(source_dir, parts[0])
        else:
            req_dir = "{}/{}".format(source_dir, req)

        if not os.path.isdir(req_dir):
            _pip_install(req, source_dir)
            installed_package = True

    return installed_package

if __name__ == '__main__':
    installed_package = main()
    data = {'installed_package': '{}'.format(installed_package).lower()}
    print json.dumps(data)

[plombardi89]

We're very interested in this feature. We're a Python and JVM shop and really do not have a ton of interest in maintaining Go source for the custom modules we want to build.

[stack72]

This was actually implemented and released by @apparentlymart :)

[deitch]

Does external in 0.8+ mean that there never will be a simple way to execute a command using local-exec and get its output saved to a var so it can be used as input to other resources?

external is great.... as long as you are willing to write a program to conform to the requirements. Sometimes, though, you want something as simple as "what is my hostname"? The simplest answer would be:

resource "null_resource" "certdir" {
  provisioner "local-exec" {
    command = "hostname"
  }
}

# and use with
${null_resource.certdir.output}
# or similar

It seems heavy overkill to have to write a wrapper to hostname that outputs json just to get the hostname. Is there a simpler solution?

[deitch]

To add to the previous comment, same thing with input. Lots of commands just expect a string on input, not necessarily json-formatted. A contrived but simple example is if I want to take some data in a template_file and convert all a into z. tr does it (as does sed, perl, etc.), but all of those work on raw strings, with no json understanding. Would be much simpler to just do:

resource "null_resource" "atoz" {
  provisioner "local-exec" {
    command = "echo ${data.template_file.some_template.rendered}" | tr 'a' 'z'"
  }
}

# and use with
${null_resource.atoz.output}
# or similar

[deitch]

And I came across the need for this again. aws_s3_bucket_object only can upload a file, not a directory. So I either need to execute it multiple times using count if I can get a list of all of the files in the directory - easy with find . -type f or - or tar/gzip it first, again easy with tar czf - ..

Both of those do a great job providing data to stdout that I could use to drive the aws_s3_bucket_object... but there is no way to get them because local-exec has no stdout, and using external requires it to have a json-only structure, which neither of those standard tools supports.

[devth]

Are there any plans to implement external resource?

[bpoland]

When I wanted the local hostname, I just used trimspace(file("/etc/hostname")) -- trimspace because the hostname file has an extra line break that I didn't want. I haven't tried it but you could probably use local-exec or https://www.terraform.io/docs/providers/local/r/file.html to write a file to the local disk, and then the file() interpolation to read back the value (might need depends_on blocks to make sure things happen in the right order).

[mtougeron]

@devth FYI, the external data source has been implemented. I've used it quite successfully in the past for stuff like this.

@bpoland The external data source should be able to provide this for you. The biggest downside will probably be that it needs the data returned in json format.

[devth]

@mtougeron thanks, I saw that too. I was referring to the 2nd part of the proposal: the external_resource resource that would implement "create", "read", "update" or "delete".

[bpoland]

Yeah I think the desire is to be able to quickly get the output from a shell command and use it. In my case, I just wanted the output from the "hostname" command but was able to read /etc/hostname instead. I didn't want to write a python script or something else just to get the local hostname.

[lorengordon]

I wrote a simple python helper for the terraform external provider. It's super simple, but it does let you just focus on the custom logic you want to implement in python.

https://gist.github.com/lorengordon/f4ceaa95b9fe669ee533a8aa40b955c1

[matti]

These external data sources get run every time - if you just want to get output once, I made a module for this https://github.com/matti/terraform-shell-resource

[rayterrill]

@apparentlymart It looks like the external_resource was only implemented as a data source, as opposed to giving the ability for create/read/update/delete. Any chance this will be extended to include CRUD capabilities? I have exactly this use case - being able to bolt-in lightweight integrations to systems where integrations do not currently exist without having to do so with Go (which I would need to learn).

[apparentlymart]

Hi @rayterrill!

The data source was implemented first in order to get a sense of how well it work work in practice, what sorts of use-cases would be implemented with it, etc. Based on that, we've seen feedback that the protocol between Terraform and the external program is too restrictive (forcing maps of strings only in both directions) and other such ergonomic problems with this approach.

Given that the external managed resource as described here would follow a similar design while adding further complexity for the resource lifecycle, we currently have the feeling that this is not the best approach and are instead making some initial steps in some different directions:

• The work currently in progress to improve the configuration language will add the possibility of string parsing functions such as jsonencode that can return arbitrarily-typed data, as an alternative to baking a particular type and encoding into a single resource. Once that is done, it is likely that a local_exec data source that just runs a program and returns its raw output would be preferable to the external data source for most cases, since it avoids the need to implement a custom interface between Terraform and command line tools. While there are no immediate plans to discontinue the external data source, feedback so far suggests that most users would prefer to use this hypothetical local_exec data source in conjunction with optional parsing functions.
• For the managed resource lifecycle there isn't really any way to avoid some Terraform-specific interface code, because most external commands are not built with a CRUD lifecycle in mind. However, the type limitations are still valid and so we decided instead to start making some progress towards supporting Terraform provider plugins written in other languages. This will take some time to get 100% working, but we'll be taking some steps in this direction shortly by switching from a Go-specific RPC prototcol to grpc. We expect some other changes will be required for the result to be totally usable, and it'll also take some effort to get similar helper functionality available in some other popular languages, but this is the long-term direction we're hoping to take instead of an external resource.

Since the multi-language provider idea will probably take some time to get to a good, usable state it is possible that we will find an interim solution similar to the local_exec data source I described above for more convenient use of external CLI tools as a resource, but we'll have to wait to think about that some more until we're past the current configuration language projects.

[rayterrill]

@apparentlymart I'm not sure I totally understand - Maybe if I provide an example of what I'm trying to accomplish, it'll help.

I'd ideally like to be able to plug in PowerShell script(s) into my Terraform configurations to handle things that are not API-enabled - I'm thinking things like AD DNS (really AD in general), custom inventory systems, etc. To do that, I'd really need some mechanism to understand create vs update vs destroy so I would know when I need to create or update something (apply), or remove something (destroy) - say a DNS CNAME in Active Directory. Would the local_exec data source allow that possibility?

Just thinking out loud - maybe some way to have external programs conform to a standard - like they would need to handle a "action" parameter or something that would tell them whether terraform was run with apply or destroy, and then some mechanism to handle the consumption of all of the attributes defined in the local_exec or external configuration entry? Maybe this already works and I just don't know how to use it?

[apparentlymart]

A possible way we could have a medium-term solution here is to have a local_exec resource where you describe an action to take for each lifecycle action:

# design idea; not currently implemented
resource "local_exec" "example" {
  create_command = ["your-program", "create", "..."]
  update_command = ["your-program", "update", "..."]
  read_command   = ["your-program", "read", "..."]
  delete_command = ["your-program", "delete", "..."]
}

The thing we'd need to figure out here is how best to export the results of such a resource. It could just be that there's a single string attribute exported that contains the raw stdout output of the most recently-run command, and it's up to the user to make sure that all of the commands produce output in a consistent format. It sounds like that would work for your situation, where you'd be writing these programs specifically with Terraform in mind.

The main difference here with external is that this resource would deal only in strings (arguments and stdout result) and actually processing those inputs and outputs would be done with features of the configuration language. Functionality like I discussed originally for a resource "external" could be achieved by using the jsonencode (already existing) and jsondecode (planned) functions, but you could also just treat the input and result as plain text or process it using other marshalling functions that might be added to the configuration language in future.

[bpoland]

^ that sounds awesome!

[rayterrill]

I'm with @bpoland - the design idea for a local_exec resource sounds awesome @apparentlymart. :)

[apparentlymart]

Thanks for the feedback! We won't be able to act on that immediately since we need to complete the configuration language project first (in particular, so that the mentioned jsondecode function can be implemented) but we'll revisit this again once we get there and see how this might look in practice.

[pdecat]

Once that is done, it is likely that a local_exec data source that just runs a program and returns its raw output would be preferable to the external data source for most cases, since it avoids the need to implement a custom interface between Terraform and command line tools.
While there are no immediate plans to discontinue the external data source, feedback so far suggests that most users would prefer to use this hypothetical local_exec data source in conjunction with optional parsing functions.

+1, I can confirm I would replace my current usage of the external data source by that one.

[matti]

What's a concrete use case for read vs create

resource "local_exec" "example" {
  create_command = ["your-program", "create", "..."]
  read_command   = ["your-program", "read", "..."]
}

Is read going to be executed on each apply? Read sounds like a data source to me.

[bpoland]

Read would be executed on every apply during the refresh stage, to make sure that the resource still exists and is configured properly. If any out of band changes are made, this allows terraform to detect and correct them.

[mikesouza]

@apparentlymart

• The work currently in progress to improve the configuration language will add the possibility of string parsing functions such as jsonencode that can return arbitrarily-typed data, as an alternative to baking a particular type and encoding into a single resource. Once that is done, it is likely that a local_exec data source that just runs a program and returns its raw output would be preferable to the external data source for most cases, since it avoids the need to implement a custom interface between Terraform and command line tools. While there are no immediate plans to discontinue the external data source, feedback so far suggests that most users would prefer to use this hypothetical local_exec data source in conjunction with optional parsing functions.

I would love to have this ability.

Due to the strictness of the external_data_source, I ultimately implemented my own provider terraform-provider-glue with a data source to do exactly this. I also implemented additional "filter" data sources which can be fed the raw output from this data source and parse/ query it by JMESPath or regular expression. It would be great to have optional functions for these.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.