Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In gitlab, "apply" pipeline hangs on plans if there are no changes, since 0.25.0 #3725

Closed
lukemassa opened this issue Aug 31, 2023 · 5 comments · Fixed by #3747
Closed

In gitlab, "apply" pipeline hangs on plans if there are no changes, since 0.25.0 #3725

lukemassa opened this issue Aug 31, 2023 · 5 comments · Fixed by #3747
Labels
bug Something isn't working

Comments

@lukemassa
Copy link
Contributor

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Since upgrading to 0.25.0, when an MR is planned in gitlab that does not have "no changes", the "apply" pipeline runs and hangs. After the user types "atlantis apply", the actual apply goes, so this seems more an aesthetic issue than anything, but it's confusing for users and also disrupts automation we have that expects there to be no pipelines running after plan finishes.

Reproduction Steps

  1. In gitlab, open an MR that makes a change
  2. Watch the plan finish, and post a comment
  3. Note that there is an indefinitely running "apply" job

Logs

Logs 
{"level":"debug","ts":"2023-08-31T21:08:26.989Z","caller":"events/events_controller.go:747","msg":"Processing...","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:26.989Z","caller":"server/middleware.go:72","msg":"POST /events – respond HTTP 200","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:27.144Z","caller":"events/project_command_builder.go:299","msg":"1 files were modified in this pull request","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.144Z","caller":"events/project_command_builder.go:341","msg":"got workspace lock","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"info","ts":"2023-08-31T21:08:27.144Z","caller":"events/working_dir.go:229","msg":"creating dir \"/atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/default\"","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:27.561Z","caller":"events/working_dir.go:261","msg":"ran: git clone --depth=1 --branch test_remove_baseline --single-branch https://atlantis-svc:<redacted>@gitlab.dev.tripadvisor.com/cloud/accounts/cloud-test-202307-sbx.git /atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/default. Output: Cloning into '/atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/default'...","json":{}}
{"level":"info","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_command_builder.go:366","msg":"successfully parsed atlantis.yaml file","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_command_builder.go:373","msg":"moduleInfo for /atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/default (matching \"\") = map[]","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_finder.go:185","msg":"found downstream projects for \"regional-baseline/us-east-1/baseline.tf\": []","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_finder.go:192","msg":"checking if project at dir \".\" workspace \"cloud-test-202307\" was modified","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_finder.go:192","msg":"checking if project at dir \"regional-baseline/us-east-1\" workspace \"cloud-test-202307-regional-baseline-us-east-1\" was modified","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_finder.go:233","msg":"file \"regional-baseline/us-east-1/baseline.tf\" matched pattern","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_finder.go:192","msg":"checking if project at dir \"us-east-1\" workspace \"cloud-test-202307-us-east-1\" was modified","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"info","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_command_builder.go:397","msg":"1 projects are to be planned based on their when_modified config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_command_builder.go:400","msg":"determining config for project at dir: \"regional-baseline/us-east-1\" workspace: \"cloud-test-202307-regional-baseline-us-east-1\"","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:301","msg":"MergeProjectCfg started","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting import_requirements: [] from default server config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting workflow: \"ta\" from repos[1], id: /gitlab\\.dev\\.tripadvisor\\.com\\/cloud\\/.*/","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting delete_source_branch_on_merge: false from default server config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting repo_locking: true from default server config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting policy_check: false from default server config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting plan_requirements: [] from default server config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting apply_requirements: [approved,mergeable,undiverged] from repos[1], id: /gitlab\\.dev\\.tripadvisor\\.com\\/cloud\\/.*/","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting allowed_overrides: [workflow,apply_requirements,plan_requirements] from repos[1], id: /gitlab\\.dev\\.tripadvisor\\.com\\/cloud\\/.*/","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:591","msg":"setting allow_custom_workflows: false from default server config","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:369","msg":"MergeProjectCfg completed","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:369","msg":"MergeProjectCfg completed","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:369","msg":"MergeProjectCfg completed","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"valid/global_cfg.go:372","msg":"final settings: plan_requirements: [], apply_requirements: [approved,mergeable,undiverged], import_requirements: [], workflow: ta","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.562Z","caller":"events/project_command_context_builder.go:98","msg":"Building project command context for plan","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.564Z","caller":"terraform/terraform_client.go:321","msg":"Found required_version setting of \">= v1.5.0\"","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.564Z","caller":"terraform/terraform_client.go:290","msg":"Listing Terraform versions available at: https://releases.hashicorp.com/terraform","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"info","ts":"2023-08-31T21:08:27.612Z","caller":"terraform/terraform_client.go:362","msg":"Detected module requires version: 1.5.6","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:27.612Z","caller":"metrics/debug.go:52","msg":"timer","json":{"name":"atlantis.builder.execution_time","value":0.622390169,"tags":{},"type":"timer"}}
{"level":"debug","ts":"2023-08-31T21:08:27.885Z","caller":"events/plan_command_runner.go:122","msg":"deleting previous plans and locks","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:28.065Z","caller":"metrics/debug.go:42","msg":"counter","json":{"name":"atlantis.builder.execution_success","value":1,"tags":{},"type":"counter"}}
{"level":"debug","ts":"2023-08-31T21:08:28.065Z","caller":"metrics/debug.go:42","msg":"counter","json":{"name":"atlantis.builder.projects","value":1,"tags":{},"type":"counter"}}
{"level":"info","ts":"2023-08-31T21:08:28.174Z","caller":"events/project_locker.go:86","msg":"acquired lock with id \"cloud/accounts/cloud-test-202307-sbx/regional-baseline/us-east-1/cloud-test-202307-regional-baseline-us-east-1\"","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:28.174Z","caller":"events/project_command_runner.go:537","msg":"acquired lock for project","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"info","ts":"2023-08-31T21:08:28.174Z","caller":"events/working_dir.go:229","msg":"creating dir \"/atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/cloud-test-202307-regional-baseline-us-east-1\"","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:28.619Z","caller":"events/working_dir.go:261","msg":"ran: git clone --depth=1 --branch test_remove_baseline --single-branch https://atlantis-svc:<redacted>@gitlab.dev.tripadvisor.com/cloud/accounts/cloud-test-202307-sbx.git /atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/cloud-test-202307-regional-baseline-us-east-1. Output: Cloning into '/atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/cloud-test-202307-regional-baseline-us-east-1'...","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:28.620Z","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/atlantis/ta-terragrunt $BASE_REPO_NAME plan\" in \"/atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/cloud-test-202307-regional-baseline-us-east-1/regional-baseline/us-east-1\"","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.othersys","value":1728590,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.alloc","value":10776024,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.total","value":1936803960,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.sys","value":32480272,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.lookups","value":0,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.malloc","value":19280386,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.frees","value":19202321,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.stack.mcache_inuse","value":9600,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.stack.mcache_sys","value":15600,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.stack.inuse","value":1572864,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.stack.sys","value":407400,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.stack.mspan_inuse","value":323568,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.heap.idle","value":9289728,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.heap.inuse","value":14303232,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.heap.released","value":8314880,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.heap.objects","value":78065,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.heap.alloc","value":10776024,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.heap.sys","value":23592960,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.gc.sys","value":5157616,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.gc.next","value":15730080,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.gc.last","value":1693516072381247500,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.gc.pause_total","value":134823588,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.memory.gc.count","value":653,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.cpu.goroutines","value":21,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:33.065Z","caller":"metrics/debug.go:47","msg":"gauge","json":{"name":"atlantis.scheduled.runtime.cpu.cgo_calls","value":0,"tags":{},"type":"gauge"}}
{"level":"debug","ts":"2023-08-31T21:08:34.292Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.195.149.52:9914","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:34.292Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:34.650Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.195.149.111:7528","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:34.650Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:35.180Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.195.149.165:46678","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:35.180Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
{"level":"info","ts":"2023-08-31T21:08:38.843Z","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/atlantis/ta-terragrunt $BASE_REPO_NAME plan\" in \"/atlantis-data/repos/cloud/accounts/cloud-test-202307-sbx/17/cloud-test-202307-regional-baseline-us-east-1/regional-baseline/us-east-1\"","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17","duration":10.223031297}}
{"level":"info","ts":"2023-08-31T21:08:39.131Z","caller":"events/instrumented_project_command_runner.go:88","msg":"plan success. output available at: https://gitlab.dev.tripadvisor.com/cloud/accounts/cloud-test-202307-sbx/-/merge_requests/17","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:39.132Z","caller":"metrics/debug.go:52","msg":"timer","json":{"name":"atlantis.project.plan.execution_time","value":11.244439636,"tags":{"base_repo":"cloud/accounts/cloud-test-202307-sbx","pr_number":"17","project":"","project_path":"regional-baseline/us-east-1","terraform_version":"1.5.6","workspace":"cloud-test-202307-regional-baseline-us-east-1"},"type":"timer"}}
{"level":"debug","ts":"2023-08-31T21:08:39.410Z","caller":"events/db_updater.go:25","msg":"updating DB with pull results","json":{"repo":"cloud/accounts/cloud-test-202307-sbx","pull":"17"}}
{"level":"debug","ts":"2023-08-31T21:08:39.669Z","caller":"server/middleware.go:45","msg":"POST /events – from 10.195.149.52:9918","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:39.669Z","caller":"events/events_controller.go:111","msg":"handling GitLab post","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:39.698Z","caller":"events/events_controller.go:497","msg":"request valid","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:39.698Z","caller":"events/events_controller.go:501","msg":"handling as comment event","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:39.698Z","caller":"events/events_controller.go:747","msg":"Ignoring non-command comment: \"Ran Plan for dir: `regional-baseline/us-...\"","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:39.698Z","caller":"server/middleware.go:72","msg":"POST /events – respond HTTP 200","json":{}}
{"level":"debug","ts":"2023-08-31T21:08:39.959Z","caller":"metrics/debug.go:52","msg":"timer","json":{"name":"atlantis.cmd.autoplan.execution_time","value":12.969040726,"tags":{},"type":"timer"}}
{"level":"debug","ts":"2023-08-31T21:08:40.065Z","caller":"metrics/debug.go:42","msg":"counter","json":{"name":"atlantis.project.plan.execution_success","value":1,"tags":{"base_repo":"cloud/accounts/cloud-test-202307-sbx","pr_number":"17","project":"","project_path":"regional-baseline/us-east-1","terraform_version":"1.5.6","workspace":"cloud-test-202307-regional-baseline-us-east-1"},"type":"counter"}}

Environment details

  • Atlantis version: 0.25.0
  • Deployment method: eks
  • If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: N/A
  • Atlantis flags:
```yaml
    ATLANTIS_AUTOMERGE: true
    ATLANTIS_PARALLEL_POOL_SIZE: 5
    ATLANTIS_QUIET_POLICY_CHECKS: true
    ATLANTIS_ENABLE_POLICY_CHECKS: false
    ATLANTIS_GITLAB_USER: atlantis-svc
    ATLANTIS_GITLAB_HOSTNAME: gitlab.dev.tripadvisor.com
    ATLANTIS_WRITE_GIT_CREDS: true

Atlantis server-side config file:

    repos:
    - id: /gitlab\.dev\.tripadvisor\.com\/cloud\/.*/
      apply_requirements: [approved, mergeable, undiverged]
      workflow: ta
      allowed_overrides: [workflow, apply_requirements, plan_requirements]
    workflows:
      default:
        plan:
          steps:
          - run: terragrunt plan -no-color -out=$PLANFILE
        apply:
          steps:
          - run: terragrunt apply -no-color $PLANFILE
      ta:
        plan:
          steps:
          - run: /home/atlantis/ta-terragrunt $BASE_REPO_NAME plan
        apply:
          steps:
          - run: /home/atlantis/ta-terragrunt $BASE_REPO_NAME apply

Repo atlantis.yaml file:

version: 3
projects:
- name: cloud-test-202307
  workspace: cloud-test-202307
  dir: .
  autoplan:
    when_modified:
    - '*.tf*'
    - terragrunt.hcl
- dir: regional-baseline/us-east-1
  workspace: cloud-test-202307-regional-baseline-us-east-1
- dir: us-east-1
  workspace: cloud-test-202307-us-east-1
  autoplan:
    when_modified:
    - '**/*.tf*'
    - '**/terragrunt.hcl'
    - access-rules.yaml
    - secrets.enc.yaml

Additional Context

This is what pipelines look like when there are changes. Note that the plan has been posted, and yet the pipeline has been stuck "running" doing nothing for several minutes (the same plan from the log output, which also went quiet after the plan was done).

with_changes

When there are no changes, it still runs the "apply" job (which I believe is different than before, but not itself a problem). However in this case, it "completes", so there is no running pipelines, and this is fine.

no_changes
@lukemassa lukemassa added the bug Something isn't working label Aug 31, 2023
@jseiser
Copy link

jseiser commented Sep 5, 2023

We are getting some gitlab strangeness as well, not sure if the issue is related.

{"level":"warn","ts":"2023-09-05T17:18:52.925Z","caller":"events/apply_command_runner.go:97","msg":"unable to update commit status: POST https://gitlab.company.network/api/v4/projects/trex/trex_ops_infra/statuses/7ef6b700af4648744198658285aed6ab4a2ae516: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via \"run\")}","json":{"repo":"trex/trex_ops_infra","pull":"89"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgit.luolix.top/runatlantis/atlantis/server/events/apply_command_runner.go:97\ngit.luolix.top/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgit.luolix.top/runatlantis/atlantis/server/events/command_runner.go:301"}

it then starts applying, gets to the 2nd of 10 applies, and just stops doing anything

{"level":"info","ts":"2023-09-05T17:19:45.073Z","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/atlantis-data/bin/terraform1.5.2 apply -input=false \\\"/atlantis-data/repos/trex/trex_ops_infra/89/default/prod/core-applications/aws-ebs-csi-driver/default.tfplan\\\"\" in \"/atlantis-data/repos/trex/trex_ops_infra/89/default/prod/core-applications/aws-ebs-csi-driver\"","json":{"repo":"trex/trex_ops_infra","pull":"89","duration":49.864978124}}
{"level":"info","ts":"2023-09-05T17:19:45.073Z","caller":"runtime/apply_step_runner.go:58","msg":"apply successful, deleting planfile","json":{"repo":"trex/trex_ops_infra","pull":"89"}}
{"level":"info","ts":"2023-09-05T17:19:45.505Z","caller":"events/instrumented_project_command_runner.go:88","msg":"apply success. output available at: https://gitlab.company.network/trex/trex_ops_infra/-/merge_requests/89","json":{"repo":"trex/trex_ops_infra","pull":"89"}}

The apply that never ends, is completed though..

helm_release.aws-efs-csi-driver: Modifying... [id=aws-efs-csi-driver]
helm_release.aws-efs-csi-driver: Still modifying... [id=aws-efs-csi-driver, 10s elapsed]
helm_release.aws-efs-csi-driver: Still modifying... [id=aws-efs-csi-driver, 20s elapsed]
helm_release.aws-efs-csi-driver: Still modifying... [id=aws-efs-csi-driver, 30s elapsed]
![Atlantis](https://github.com/runatlantis/atlantis/assets/4855527/27e84cef-ca07-4f67-a642-63078ce92eb2)

helm_release.aws-efs-csi-driver: Still modifying... [id=aws-efs-csi-driver, 40s elapsed]
helm_release.aws-efs-csi-driver: Still modifying... [id=aws-efs-csi-driver, 50s elapsed]
helm_release.aws-efs-csi-driver: Modifications complete after 52s [id=aws-efs-csi-driver]

This is on the Atlantis 0.25.0

@jseiser
Copy link

jseiser commented Sep 7, 2023 •
edited
Loading

We are getting the same, or a very similar issue. The apply pipeline just hangs forever, even when its not doing anything. These are all applies that show they never completed, but most are actually from atlantis plan

AllPlansCompleted

Atlantis-PLans

@lukemassa lukemassa mentioned this issue Sep 7, 2023
Merged
@lukemassa
Copy link
Contributor Author

I think the above PR might fix the issue, I still have to figure out how to test it on my side though

@jamengual
Copy link
Contributor

jamengual commented Sep 7, 2023 via email

@lukemassa
Copy link
Contributor Author

lukemassa commented Sep 7, 2023 •
edited
Loading

I was actually just able to test out my change in #3747, which at least fixes the specific problem I reported.

@lukemassa lukemassa mentioned this issue Sep 13, 2023
Open
@GenPage GenPage mentioned this issue Sep 25, 2023
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Do not unnecessarily update apply check if it doesn't exist yet lukemassa/atlantis
3 participants
@jamengual @lukemassa @jseiser