Allow per-command Environment selection (eg, with an environment variable)

[glasser]

We currently use Terraform 0.7.3 and are looking to upgrade to 0.9. One thing we do is run a set of slow end-to-end tests that you run against your Terraform configuration that builds a whole test environment based on the same configuration, runs some tests against the servers, and tears it down.

It's important that we can continue working on our development environment in the same directory simultaneously with running these tests.

Right now we do this by always running TF with a wrapper script that uses my PR #8720 to separate these setups into different .terraform directories.

However, we would love to move away from wrapper scripts and use the new Environment feature, which is designed exactly for this purpose. Developers would each create an environment with their name using terraform env new $USER, and when they run the end-to-end tests, the test suite would generate an environment name and run terraform using that environment.

The problem is that there's no non-racy way to run terraform in multiple environments on the same data directory using the environment feature, because the only way to specify which environment to use is via the filesystem. Having the test suite run terraform env select before and after every command isn't good enough, because the main command it runs may be runs slow and I may want to concurrently run on my development environment while the command runs.

I'm pretty sure I can get exactly what I want by adding a TF_ENVIRONMENT environment variable, which is checked before the .terraform/environment file. I intend to do that for my company's use and file a PR. I am curious if any Terraform developer believes that this would be sufficient to achieve my goals or if there are other uses of the .terraform directory that are not separated by environment.

[apparentlymart]

Hi @glasser!

In principle this sounds like it would work, and seems like a reasonable idea. If you're willing to work on it and share what you learn, that'd be great!

Some things to think about:

• Modules are downloaded into .terraform/modules when you run terraform init, so your developers would need to take care not to add, remove or modify modules while the tests are running in order to get consistent results.
• In principle state locking should protect against collisions from concurrently running Terraform instances, but that will depend on any limitations of the backend you are using.
• Some Terraform resources write to the local filesystem, making it unsafe to concurrently run Terraform twice from the same directory. However, these are pretty rare and easily avoided.

Given that it's easy to set environment variables in your shell and forget about them, I think it'll be important to add some new output to terraform env list in the case where the environment variable is set, so it's clear to the user that it's being overridden:

$ terraform env list
  default
* matkins

The active environment is being overridden using the TF_ENVIRONMENT environment variable.

Should probably also add a helpful error message to terraform env select, making it refuse to work when the environment variable is overriding it:

$ terraform env select default
The environment is currently overridden using the TF_ENVIRONMENT environment variable.

To select a new environment, either update this environment variable or unset it and then
run this command again.

[glasser]

Hmm, my understanding is that you don't even need to run terraform init when you're just switching environments, right?

Good point re modules. We only use local (relative path) modules and basically try to pretend that the thing where Terraform sets up symlinks in hidden places instead of just interpreting the paths directly doesn't exist. (Our current shell scripts run terraform get before every command but I think we can stop doing that. We only use 2 or 3 modules.)

Those suggestions for terraform env make a lot of sense and I hadn't considered them!

By the way, do you agree with me that currentStateName() in backend/local/backend.go is dead code that should be deleted rather than live code that needs to be updated for this feature?

[glasser]

Also, one other question: if doing this, is it OK to skip running terraform state new? It looks to me like the state creation logic that the call to b.State(newEnv) in terraform state new runs is also run whenever you use the state for the first time elsewhere, so I'd hope the answer is yes.

[apparentlymart]

@glasser for these details I think it's better for me to bring in @jbardin to this discussion, since he worked on all this stuff and so knows better the whys and hows of the code as it's currently written. He's out of the office right now but I'll ping him about this once he's back next week.

[glasser]

Opened PR #14602.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.