Unable to override required_version terraform version using a flag

[ekhaydarov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

A new atlantis feature automatically tries to manage terraform versions for you, downloads the latest version for you and runs that instead. For repos that have floating version requirements like >=1.3 that means atlantis will try to download and use 1.4.0 for you and run that even if your state file is in 1.3.9. this should have been documented as a breaking change

The issue occurs when you have a policy check after a plan. The plan can run in tf 1.3.9 but the policy_check will run in a different terraform version, in this case 1.4.0 was released so atlantis tries to be clever and use the latest version. Of course that fails because the state file is in tf 1.3.9.

Looking at the docs I have tried setting the default version with these envs but they do nothing for the policy_check part.

  ATLANTIS_DEFAULT_TF_VERSION: 1.3.9
  DEFAULT_TERRAFORM_VERSION: 1.3.9

What does one need to do in order to turn off this "feature" completely and let us control what terraform version is being run? If not possible what flag / option is there to override the tf version to make sure that a consistent and same version is used at all times in all places?

Reproduction Steps

• your atlantis.yaml has an open required_version like >=1.3.
• use not the latest version of terraform to create an initial plan.
• let altantis do the rest

Environment details

• Atlantis version: 0.22.3
• Deployment method: helm
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: no
• Atlantis flags:

  ATLANTIS_AUTOMERGE: true
  ATLANTIS_CHECKOUT_STRATEGY: merge
  ATLANTIS_ENABLE_POLICY_CHECKS: true
  ATLANTIS_QUIET_POLICY_CHECKS: true
  ATLANTIS_ENABLE_REGEXP_CMD: true
  ATLANTIS_SILENCE_NO_PROJECTS: true
  ATLANTIS_DEFAULT_TF_VERSION: 1.3.9
  DEFAULT_TERRAFORM_VERSION: 1.3.9

Atlantis server-side config file:

repos:
- id: /.*/
  apply_requirements: [mergeable]
  workflow: default
  allowed_overrides: []
  allow_custom_workflows: false
workflows:
  default:
    plan:
      steps:
      - env:
          name: TF_WORKSPACE
          value: ""
      - env:
          name: TF_IN_AUTOMATION
          value: "true"
      - env:
          name: TF_PLUGIN_CACHE_DIR
          command: 'echo "$ATLANTIS_DATA_DIR/plugin-cache"'
      - run: terragrunt init
      - run: |
          terraform plan -input=false -refresh -no-color -lock=false -parallelism=5 $(eval echo ${COMMENT_ARGS}) -out $PLANFILE 2>&1 > out.log
          exitCode=$?
          cat out.log | grep -v 'Refreshing state...' | sed 's/^  +/+/' | sed 's/^  ~/~/' | sed 's/^  -/-/'
          exit $exitCode

    policy_check:
      steps:
      - env:
          name: TF_WORKSPACE
          value: ""
      - env:
          name: ATLANTIS_POLICIES_DIR
          command: 'echo "${DIR%$REPO_REL_DIR}policies/"'
      - env:
          name: ATLANTIS_CURRENT_DIR
          command: 'echo "${DIR}"'
      - show
      - policy_check:
          extra_args: ["--policy ${ATLANTIS_POLICIES_DIR} --all-namespaces"]

Repo atlantis.yaml file: its 16k lines. can provide relevant info if needed

[AlessioCasco]

We saw this issue also on v0.23.2

[nitrocode]

What you probably want to use is ~> 1.3.0 or = 1.3.9 for required_version.

https://www.runatlantis.io/docs/terraform-versions.html#via-terraform-config

[ekhaydarov]

that breaks original funcitonality where >=1.3 does not impact what version of terraform runs. That is internal terraform code and should have no impact on what atlantis does. That part of terraform code should be independent of what atlantis does or at the very least allow us to turn off this funcitonality.

The bug is that default terraform version does not do anything to help out.

[jamengual]

can you show your atlantis.yaml? just one project is fine.

This does not break functionality; we match the same way TF does version requirements.

how are you defining your provider versions?

[nitrocode]

I think he's saying that he wants to set a global default terraform version using ATLANTIS_DEFAULT_TF_VERSION in spite of the required_version.

Whereas the current workflow in atlantis is to override the default terraform version with required_version.

I personally do not agree with this suggestion.

Plus you can use a custom workflow to run any terraform version you like. You can hard code it if you like.

      - run: |
          terraform1.3.9 plan

[ekhaydarov]

I apologise for my lack of understanding here. I am also having a hard time seeing why atlantis needs to manage terraform versions for you? Thats outside the scope of the tool is it not?

Consider most CI workflows are all running on pretty much custom images that have all their inputs controlled. i.e. we set the atlantis version in our docker image, we set the terragrunt version, we set the terraform version. If atlantis randomly starts using differnt version of terraform why doesnt atlantis just update itself at this point and use latest for everything? I am not understanding why atlantis needs to control the tf version for you? end users need that right in normal workflows, not only in custom ones surely?

Secondly with overriding tf version through required_version. There are cases when project specific pinning needs to happen in cases like tf version only needs to be above 1.3 since they implemented a new feature for optional variable defaults. That lets our teams testing locally know that any version above 1.3 will be fine. Why would atlantis need to look at that and try to be smart for you pretty much all Ci's are custom images where teams tightly control all their versions of tools that go into that image. This atlantis change forces us to pin exact version all the time and all developers will at all times need to match required version.

I think he's saying that he wants to set a global default terraform version using ATLANTIS_DEFAULT_TF_VERSION in spite of the required_version.

Weve clearly hit some issue here though. I have set the Default version, so the plan perhaps picks up the version 1.3.9. the policy check picks up 1.4.0. If I do not set the default version, the plan tries to bump to v1.4.0 when there is no need for us to do that right now, our CI image has no clue about 1.4.0, were sitting there thinking what in our dockerfile managed to put tf 1.4.0 in there. Meanwhile our own tf installation is entirely redundant now.

So if we have a fork of terraform we cannot run it unless its a custom workflow? I apologies but i fail to see how this is not a breaking change? My atlantis.yaml needs a rerite to make our forked tf work now. thats a breaking change surely?

I am simply asking for a contingency where a user can simply use a specified tf version in all usecases. Currently its inconsistent, do you not agree? Why would required_version need to be hard coupled to tf version atlantis chooses, they should not be linked at all and managing tf version is not atlantis' responsibility IMO. Again i may be lacknig understanding and appreciate if there are other reasons for this, but I am failing to see that.

A terraform_version specified in the atlantis.yaml file takes precedence over both the --default-tf-version flag and the required_version in the terraform hcl.

So setting that would allow the policy check command to use 1.3.9? if I set tf 1.2.0 will atlantis go an download that for me? and use that in call instances? If that is the case I did not parse the docs correctly and this issue can be closed

But also if that is the case i dont understand what these variables are for and they dont work as their naming implies. They clearly dont default for all commands. They are confusing as there are two of those variables. Differences are confusing to end users, why is there a need for both? Can they be just be removed? Setting tf version explicitly though atlantis yaml and implicitly through required_version is enough. Why provide extra variables that dont actually do what you want?

  ATLANTIS_DEFAULT_TF_VERSION: 1.3.9
  DEFAULT_TERRAFORM_VERSION: 1.3.9

I hope this is constructive feedback and it doesnt feel like an atttack. Just would like to get better understanding
Thanks for the OS work and trying to help

[jamengual]

@ekhaydarov I hear you and I understand the frustration so let's try to find a solution.

how long have you used atlantis, specifically what version?

Atlantis downloads the TF version specified in required_version if is not on the base image and that has been around for a while.

On my terraforms project in this company and others we always have a version.tf like so :

terraform {
  required_version = ">= 1.2.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.0"
    }
  }
}

and in some cases, we only allow one version only, especially if is a monorepo and in that case the atlantis.yaml and the version.tf looks the same as for the required_version setting.

[nitrocode]

Perhaps we need a new flag such as this?

ATLANTIS_AUTODOWNLOAD_TERRAFORM=false

It can default to true unless specified as false. By enabling this potential flag, atlantis will stop reading the required_version argument and stop downloading the latest terraform that matches that argument.

[ekhaydarov]

Thanks for understanding and suggesting a workaround. Just double checking that when autodownload is set to false then when commands are run they simply reference terraform instead of the full path /atlantis-data/bin/terraform1.4.0, or however, it worked previously to pick non atlantis installation of terraform

[cep21]

Thank you for the ticket @ekhaydarov . We ran into the same issue! I only noticed when my atlantis apply failed because the plan step had 1.4.0 and the apply step was trying to use 1.4.1.

[steveizzle]

Thank you for the ticket @ekhaydarov . We ran into the same issue! I only noticed when my atlantis apply failed because the plan step had 1.4.0 and the apply step was trying to use 1.4.1.

Same here today. We use atlantis since quite some time and this change was very surprising. I also tried the different environment variables to pin a specific terraform version and also used required_version >= x.y.z like @ekhaydarov . I would say this is very common and totally agree that setting required_version like that should not influence atlantis in a way that the latest version should be applied automatically.

[steveizzle]

So setting that would allow the policy check command to use 1.3.9? if I set tf 1.2.0 will atlantis go an download that for me? and use that in call instances? If that is the case I did not parse the docs correctly and this issue can be closed

It seems to be correct that the terraform_version in atlantis.yaml takes precedence over everything else. In my case that would mean setting it 77 times (because this key is only available on project level). That would work as workaround for this issue, but maybe there should also be a global key to set this for all projects...

But more important! I tried to constrain the required_version on one module like that: required_version = ~> 1.3.0
In this case atlantis does not use 1.3.X, but also latest and this leads the plans/applies to fail!

[nitrocode]

That is indeed odd. For me, I've been able to use required_version = "~> 1.3.0" and have it match 1.3.9 successfully.

If you folks think we should have a new flag to respect the atlantis.yaml terraform_version and/or the --default-tf-version to take precedence, then please submit a PR. We would happily review it.

@steveizzle The atlantis version will need to be 0.23.0 or later for the required_version to work as you expect. See pr #3153.

[nitrocode]

Another flag that could be added to allow downloading but give it a maximum version so there is a central place where the tf version can be controlled

ATLANTIS_AUTODOWNLOAD_TERRAFORM_MAX="1.3.9"
ATLANTIS_AUTODOWNLOAD_TERRAFORM_MIN="1.0.0"

Both of the above could default to empty strings to maintain the current workflow

[endriu0]

Just to add to the discussion - we've used Atlantis for a while and up until recently using atlantis with below req version:

terraform {
  required_version = ">= 0.12"
}

Would give you the tf version from ATLANTIS_DEFAULT_TF_VERSION which was 0.12.31 in our case. (Old stacks not many people have time to update)

With atlantis upgrade this now defaults to latest TF which breaks a lot of our projects. Some option to keep old functionality in place would be good.

Setting req version to specific still worked like so :

terraform {
  required_version = "= 1.2.3"
}

So are we saying the old behaviour worked due to a bug that was fixed with : #3153

[nitrocode]

cc @zhuoyang author of #3153

[zhuoyang]

Just to add to the discussion - we've used Atlantis for a while and up until recently using atlantis with below req version:

terraform {
  required_version = ">= 0.12"
}

Would give you the tf version from ATLANTIS_DEFAULT_TF_VERSION which was 0.12.31 in our case. (Old stacks not many people have time to update)

With atlantis upgrade this now defaults to latest TF which breaks a lot of our projects. Some option to keep old functionality in place would be good.

Setting req version to specific still worked like so :

terraform {
  required_version = "= 1.2.3"
}

So are we saying the old behaviour worked due to a bug that was fixed with : #3153

No, the old behaviour worked due to the required_version not being respected before atlantis version 0.23.0 0.21.0 (docs)
EDIT: wrong version

[ekhaydarov]

It is clearly explained above that atlantis is not a terraform version manager. To summarise:

• There are many cases explained where end user wants to control their terraform version
• Atlantis also doesnt respect version when using policy checks
• Using a custom workflow everywhere just to make everything the same is not feasible, and the fact everything is not the same is a bug IMO

Now you are stating that atlantis was broken before 0.23.0 which adds to my opinion about this being a breaking change and needing to be announced as one. We are going round in circles now. Happy for this issue to be muted/closed and myself or someone with a more imperative need implement @nitrocode suggestion they outlined below:

That is indeed odd. For me, I've been able to use required_version = "~> 1.3.0" and have it match 1.3.9 successfully.

If you folks think we should have a new flag to respect the atlantis.yaml terraform_version and/or the --default-tf-version to take precedence, then please submit a PR. We would happily review it.

[qibint]

Facing the same issue.
We have terraform provider block defined:

terraform {
  required_version = ">= 0.12"
}

We have ATLANTIS_DEFAULT_TF_VERSION set to 1.2.1. Now with the latest image 0.24.4, the atlantis plan always picks the 1.5.3 terraform version and ignore the tf version setting which broke all our existing projects planning.

[jamengual]

Feel free to push a PR if anyone wants to work on this fix.
The old behavior was expected to work but at some point but it didn't and then the supposed fix forced the new version in, and the ATLANTIS_DEFAULT_TF_VERSION is not respected which it should.

[willswinson]

Is this related to why I am getting this error despite me setting the required_version in terraform? After I got this I also set the environment variable ATLANTIS_DEFAULT_TF_VERSION in the container and it's still not working.

Error: Unsupported Terraform Core version
│ 
│   on versions.tf line 2, in terraform:
│    2:   required_version = "1.3.7"
│ 
│ This configuration does not support Terraform version 1.5.5. To proceed,
│ either choose another supported Terraform version or update this version
│ constraint. Version constraints are normally set for good reason, so
│ updating the constraint may lead to other errors or unexpected behavior.

[jamengual]

yes.

…

On Fri, Aug 25, 2023 at 9:58 PM Will Swinson ***@***.***> wrote: Is this related to why I am getting this error despite me setting the required_version in terraform? After I got this I also set the environment variable ATLANTIS_DEFAULT_TF_VERSION in the container and it's still not working. Error: Unsupported Terraform Core version │ │ on versions.tf line 2, in terraform: │ 2: required_version = "1.3.7" │ │ This configuration does not support Terraform version 1.5.5. To proceed, │ either choose another supported Terraform version or update this version │ constraint. Version constraints are normally set for good reason, so │ updating the constraint may lead to other errors or unexpected behavior. — Reply to this email directly, view it on GitHub <#3199 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERGDF5TQ24QSGMQTAZLXXF67XANCNFSM6AAAAAAVWG2ZNI> . You are receiving this because you commented.Message ID: ***@***.***>

[WJay-tec]

I am facing the same issue as @willswinson , it seems to only affect steps that are using the run command.

workflows:
   dev:
     plan:
       steps:
       - init:
       - plan:
           extra_args: ["-var-file", "dev.tfvars"]
   stg:
     plan:
       steps:
       - init:
       - plan:
           extra_args: ["-var-file", "stg.tfvars"]
   prd:
     plan:
       steps:
       - init:
       - plan:
           extra_args: ["-var-file", "prd.tfvars"]
   
   make:
     plan:
       steps: 
       - run: make init
       - run: make plan
     apply:
       steps:
       - run: make apply

Is there any workaround for this at the moment? @jamengual

[jamengual]

the workaround is to set the terraform_version at the workflow level, you can override the default workflow if you are not setting workflows

[WJay-tec]

@jamengual like what is mentioned here?
#3199 (comment)

[jamengual]

yes

[jamengual]

who is willing to make a PR to solve this?