Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GitLab 400 Error on apply #1356

Closed
stafot opened this issue Jan 19, 2021 · 40 comments · Fixed by #1971
Closed

GitLab 400 Error on apply #1356

stafot opened this issue Jan 19, 2021 · 40 comments · Fixed by #1971
Labels
provider/gitlab

Comments

@stafot
Copy link

stafot commented Jan 19, 2021

I’m finding the plan works fine, but Atlantis logs the following warning on the subsequent apply: Unable to update commit status: POST https://gitlab[redacted]: 400 {message: Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via "enqueue")}

Helm chart version: 3.12.4
VCS: Gitlab

Result: The pipeline gets stuck to pending forever.

ref. runatlantis/helm-charts#27
@Platou
Copy link

Platou commented Mar 2, 2021

Same here, any updates? @stafot

@stafot
Copy link
Author

stafot commented Mar 2, 2021

Nope to be honest.

@jamengual
Copy link
Contributor

those are comments on your PR or messages in the log on the Atlantis server?

@stafot
Copy link
Author

stafot commented Mar 3, 2021 •
edited
Loading

@jamengual Exactly.
, the logs on the atlantis pod

@jamengual
Copy link
Contributor

and does atlantis still works?

@stafot
Copy link
Author

stafot commented Mar 3, 2021

Yes atlantis plan works fine. I am getting this when I comment, atlantis apply

@jamengual
Copy link
Contributor

Then My guess the gitlab api got updated and now this "enqueue" thing is in deprecating mode.

@Platou
Copy link

Platou commented Mar 3, 2021

Good guess, I have the issue since updating to Gitlab 13.9.1

@stafot
Copy link
Author

stafot commented Mar 3, 2021

I had the issue with 13.6.x. Today upgraded to 13.9.1 but didn't retry

@stafot
Copy link
Author

stafot commented Mar 6, 2021

Did you have any luck resolving it @Platou ?

@Platou
Copy link

Platou commented Mar 6, 2021

Did you have any luck resolving it @Platou ?

I think it stopped doing this... I’ll have to double check

@esn89
Copy link

esn89 commented Mar 10, 2021

I'm at 13.9 and I an experiencing this problem. Atlantis plan works, but apply does not. I can only apply via "-p" option.

@stafot
Copy link
Author

stafot commented Mar 12, 2021

I also upgrade to latest Gitlab and the problem persists.

@stafot
Copy link
Author

stafot commented Mar 12, 2021

@esn89 could you elaborate more the -p option?

@esn89
Copy link

esn89 commented Mar 12, 2021

applying things per project. so for example:

atlantis apply -p nameofproject1
atlantis apply -p nameofproject2

@spirosoik
Copy link
Contributor

@esn89 we did some troubleshooting with @stafot

The problem seems to happen when you try to use -out flag and put the plan outside of the working copy of atlantis (the directory which atlantis clones the repo etc.). So if you remove the -out flag it will work.

Atlantis has a prerequisite to find in the working directory the pending plan. Here is the code for this https://github.com/runatlantis/atlantis/blob/master/server/events/project_command_builder.go#L361-L389

@esn89
Copy link

esn89 commented Mar 13, 2021

@spirosoik

Thanks for your reply. Right now, I am not using any -out flags at all. I just have a simple terraform plan -no-color running.

Is it possible for you to post your atlantis.yaml?

@spirosoik
Copy link
Contributor

spirosoik commented Mar 15, 2021 •
edited
Loading

@esn89 The atlantis.yaml in the terraform repo:

version: 3
automerge: true
projects:
- dir: .
  apply_requirements: [approved, mergeable]
  workflow: core

The server repo config for atlantis:

repos:
- id: /.*/
  apply_requirements: ["approved", "mergeable"]
  allowed_overrides: [apply_requirements, workflow]
  allow_custom_workflows: true
workflows:
  default:
    plan:
      steps: [init, plan]
    apply:
      steps: [apply]
  core:
    plan:
      steps:
      - run: AWS_PROFILE="cloud-team" terraform init
      - run: AWS_PROFILE="cloud-team" terraform plan
    apply:
      steps:
      - run: AWS_PROFILE="cloud-team" terraform apply

@AJakkam38
Copy link

Any fix for this?
We are still facing this issue on both default and custom workflows.

Helm chart version: 3.12.11
VCS: Gitlab - 13.11.5

@alexandrst88
Copy link

I had a theory, how to reproduce this one. I have a MR with 3 dirs to plan/apply. 1 of them is directory where I removed tf and terragrunt.hcl files. Atlantis is expected apply on dir with deleted files. So even i run plan + apply on 2 dirs with tf files, atlantis anyway expects to run apply and send "pending" to gitlab commits status api call. Based on https://gitlab.com/gitlab-org/gitlab-foss/-/issues/25807. When atlantis sends "pending" event, its get queued on some internal gitlab's queue and rejects.

@josephlim75
Copy link

josephlim75 commented Oct 25, 2021 •
edited
Loading

I am experiencing a similar issue with Atlantis version 0.17.2 custom workflow

repos.yaml

repos:
- id: "/.*/"
  workflow: epsoft-terraform
  allowed_overrides: [apply_requirements, workflow]
  allow_custom_workflows: true
  allowed_workflows: [custom-terraform]
workflows:
  custom-terraform:
    plan:
      steps:
      - run: /opt/atlantis/bin/tf-pre-init.sh
      - run: /opt/tfenv/bin/terraform init -no-color -backend-config=backend.tfvars
      - run: /opt/tfenv/bin/terraform plan -no-color
    apply:
      steps:
      - run: /opt/tfenv/bin/terraform apply -no-color -auto-approve

plan workflow works without issue but apply has the following error from atlantis

Error

{
  "level": "warn",
  "ts": "2021-10-25T13:53:45.765Z",
  "caller": "events/apply_command_runner.go:199",
  "msg": "unable to update commit status: POST https://<redacte>/api/v4/projects/cloud-infrastructure/terraform-google/projects/cloud-project-test/statuses/786f48c2cfd2bdc6b8fcb6cafdcdf427aab27c69b: 400 {message: Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via \"enqueue\")}",
  "json": {
    "repo": "cloud-infrastructure/terraform-google/projects/cloud-project-test",
    "pull": "2"
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).updateCommitStatus\n\t/go/src/github.com/runatlantis/atlantis/server/events/apply_command_runner.go:199\ngit.luolix.top/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\t/go/src/github.com/runatlantis/atlantis/server/events/apply_command_runner.go:158\ngit.luolix.top/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/go/src/github.com/runatlantis/atlantis/server/events/command_runner.go:212"
}

@josephlim75
Copy link

This issue has been open for more than 6 months, did anyone find the solution to this problem?

@josephlim75
Copy link

If i remove all the custom workflow from repos.yaml, using the default workflow, it works. This does not solve my problem because I need a custom workflow to work

@jamengual
Copy link
Contributor

are you using -out flag?

#1356 (comment)

@josephlim75
Copy link

josephlim75 commented Oct 26, 2021 •
edited
Loading

No -out flag. If I just have my repos.yaml below without the custom workflow, it works

repos:
- id: "/.*/"
  pre_workflow_hooks:
    - run: /opt/epsoft/terraform/atlantis/tf-pre-init.sh

And this is how i start my atlantis server

./atlantis server \
--repo-config=/etc/atlantis/repos.yaml \
--gitlab-hostname="<redacted>" \
--atlantis-url="<redacted>" \
--gitlab-user="<redacted>" \
--gitlab-token="<redacted>" \
--gitlab-webhook-secret="<redacted>" \
--repo-allowlist="*"

@jamengual
Copy link
Contributor

jamengual commented Oct 26, 2021 via email

@josephlim75
Copy link

josephlim75 commented Oct 26, 2021 •
edited
Loading

Nope, my tf-pre-init.sh is just to checkout other repos because my terraform scripts are relying some of the configuration. The script is purely checking if the external repo is not clone, make sure it clone first before running the terraform script.

Below is just a part of the script but the main purpose is to clone additional repository.

#!/bin/bash

ATLANTIS_SHARED_REPOS=/atlantis/.atlantis/repos/.....

clone_repo() {
  local repo=$1

  rm -rf  ${ATLANTIS_SHARED_REPOS}/${repo}
  git clone https://redacted/${repo}.git ${ATLANTIS_SHARED_REPOS}/${repo}
}

clone_repo "cloud-init"
clone_repo "modules"

@jamengual
Copy link
Contributor

jamengual commented Oct 26, 2021 via email

@josephlim75
Copy link

josephlim75 commented Oct 26, 2021 •
edited
Loading

The latest 0.17.4 binary is not downloadable. Doesn't seems to have build the binary. I can try 0.17.3

The following link is not working, but works for the rest of version
https://github.com/runatlantis/atlantis/releases/download/v0.17.4/atlantis_linux_amd64.zip

@jamengual
Copy link
Contributor

jamengual commented Oct 26, 2021 via email

@erickfaustino
Copy link

Hi!
I'm having this problem right now using it with gitlab.com
My plan work, but when I try to apply, I got this in the PR:

Ran Apply for dir: global/aws_organization workspace: global_aws_organization
Found no template. This is a bug!

Also, I can see this log inside the atlantis container:

{"level":"info","ts":"2021-10-27T20:23:50.160Z","caller":"events/events_controller.go:417","msg":"parsed comment as command=\"apply\" verbose=true dir=\"us-east-1/vpc/dev\" workspace=\"us-east-1_vpc_dev\" project=\"\" flags=\"\"","json":{}}
{"level":"info","ts":"2021-10-27T20:23:50.823Z","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"terragrunt-atlantis-config generate --output atlantis.yaml --autoplan --parallel --create-workspace\" in \"/home/atlantis/.atlantis/repos/memed/sre/terraform-foundation/10/default\"","json":{"repo":"memed/sre/terraform-foundation","pull":"10"}}
{"level":"warn","ts":"2021-10-27T20:23:51.172Z","caller":"events/apply_command_runner.go:94","msg":"unable to update commit status: POST https://gitlab.com/api/v4/projects/memed/sre/terraform-foundation/statuses/e3aaf069b48ad1e5cf9a7c1acb266a9554949be6: 400 {message: Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via \"enqueue\")}","json":{"repo":"memed/sre/terraform-foundation","pull":"10"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\t/home/circleci/project/server/events/apply_command_runner.go:94\ngit.luolix.top/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/home/circleci/project/server/events/command_runner.go:214"}
{"level":"info","ts":"2021-10-27T20:23:52.212Z","caller":"events/apply_command_runner.go:110","msg":"pull request mergeable status: true","json":{"repo":"memed/sre/terraform-foundation","pull":"10"}}
{"level":"info","ts":"2021-10-27T20:23:52.213Z","caller":"events/project_command_context_builder.go:240","msg":"cannot determine which version to use from terraform configuration, detected 0 possibilities.","json":{"repo":"memed/sre/terraform-foundation","pull":"10"}}
{"level":"info","ts":"2021-10-27T20:23:52.213Z","caller":"events/apply_command_runner.go:141","msg":"Running applies in parallel","json":{"repo":"memed/sre/terraform-foundation","pull":"10"}}

I wonder if it has something to do with terraform/terragrunt versions, or if is directly related to the message:
400 {message: Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via \"enqueue\")}

@jamengual
Copy link
Contributor

jamengual commented Oct 27, 2021 via email

@josephlim75
Copy link

josephlim75 commented Oct 28, 2021 •
edited
Loading

Honestly, I'm not sure the issue is at gitlab or atlantis. I have changed my repos.yaml not to use custom apply workflow, and everything just works. Below is my final configuration that works on plan and apply. As long as I use the default apply and not a custom one, it works as expected.

repos:
- id: "/.*/"
workflows:
  default:
    plan:
      steps:
      - run: /opt/atlantis/tf-pre-hook.sh
      - run: /opt/atlantis/tf-init.sh -no-color
      - plan
    apply:
      steps: [apply]

@ivanilves
Copy link

💡 Hope it will be helpful, I've fixed my custom workflow by generating a plan file (-out) with name expected by Atlantis.

In case of shell:

PLAN_FILE=$(pwd | awk -F"/" '{print $NF}')-default.tfplan

and Makefile:

PLAN_FILE := $(shell pwd | awk -F"/" '{print $$NF}')-default.tfplan

The minimalistic Makefile would look like this:

TERRAFORM = terraform${ATLANTIS_TERRAFORM_VERSION}

export TF_IN_AUTOMATION := true

PLAN_FILE := $(shell pwd | awk -F"/" '{print $$NF}')-default.tfplan

init:
	@${TERRAFORM} init -no-color

plan:
	@${TERRAFORM} plan -no-color -out ${PLAN_FILE}

apply:
	@${TERRAFORM} apply -no-color ${PLAN_FILE}
  • Before this change, atlantis apply -p <project> was working, though "wildcard" atlantis apply was not.
  • After this change atlantis apply started to work as well, as Atlantis started to find plan files successfully 🎉

@esn89
Copy link

esn89 commented Nov 5, 2021

@ivanilves
Still no luck. What version of atlantis are you using?

Also, for your $PLAN_FILE value, what is it supposed to be?

@joncolby
Copy link

joncolby commented Dec 11, 2021 •
edited
Loading

I have the same issue, atlantis v0.17.5 . In my case I am running custom workflows with terragrunt.

gitlab version GitLab Enterprise Edition 13.12.8-ee

{"level":"warn","ts":"2021-12-09T15:19:13.379Z","caller":"events/apply_command_runner.go:94","msg":"unable to update commit status: POST https://somegitlabserver/api/v4/projects/myproject/statuses/5a7c6bbddb2f1e99e3e2936e66bd76a5a3a7a920: 400 {message: Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via \"enqueue\")}","json":{"repo":"some/gitlab/repo/project","pull":"145"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\t/home/circleci/project/server/events/apply_command_runner.go:94\ngit.luolix.top/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/home/circleci/project/server/events/command_runner.go:214"}

@jamengual
Copy link
Contributor

jamengual commented Dec 11, 2021 via email

@lkysow lkysow changed the title Apply is not working GitLab 400 Error on pply Dec 15, 2021
@lkysow lkysow changed the title GitLab 400 Error on pply GitLab 400 Error on apply Dec 15, 2021
@jamengual
Copy link
Contributor

does anyone have this issue constantly to the point that renders Atlantis basically useless?
there seem to be multiple different issues commented in this issue which was originally open about the 400 error issue.

if someone has the time to be more proactive in debugging this, maybe we can offer some help but without that, it is impossible for us to replicate the exact scenario.

@syphernl
Copy link
Contributor

syphernl commented Jan 4, 2022

My GitLab MR was stuck in pending (preventing me to merge it). I came across this issue and did some digging.

The problem here is that the status is trying to go from pending to success which is not allowed.
It needs to go trough running first before it can go to success. Optionally the whole pending part can be skipped and replaced with running instead.

I fixed that by changing it to running first:

curl --request POST --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "https://gitlab.com/api/v4/projects/PROJECTID/statuses/COMMITSHA?state=running&name=atlantis%2Fapply"

And then to success

curl --request POST --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" "https://gitlab.com/api/v4/projects/PROJECTID/statuses/COMMITSHA?state=success&name=atlantis%2Fapply"

ISTM if the pending state is replaced with running this problem should be resolved. I am however not clear as to whether a pending state is needed or not..

@syphernl syphernl mentioned this issue Jan 4, 2022
Merged
@jamengual
Copy link
Contributor

People here should try @syphernl fix and report back: #1971

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
provider/gitlab
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(gitlab-client): change pending to running state syphernl/atlantis