gitlab: atlantis/apply job stays pending

[syphernl]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

We're using Atlantis with GitLab. If we do partial applies using comments the hover-text on the atlantis/apply job stays at 0/3 projects applied successfully and this stays pending.

Since there is also an actual atlantis/plan: xxxx for the individual project the individual atlantis/apply job seems redundant.

The atlantis/apply job itself is the only one without a linked job URL.

Reproduction Steps

• Run a plan for a multi-project
• Do manual applies for 2 out of 3
• See that the atlantis/apply -d xxxx -w xxxx remains at 0 of 3 projects applied and stays in pending.

Logs

Environment details

If not already included, please provide the following:

• Atlantis version: 0.19.1

Additional Context

[cilindrox]

Ran into this slightly differently on GitHub: the apply step remains as pending and having the automerge enabled triggers a 405 Required status check "atlantis/plan" is failing. error on GH's API (expected, since the apply is a merge requirement).

From the looks of it, there's some sort of missing post-plan status update missing or failing to trigger.

[ddrugeon]

Hello,
I have the same issue with my Atlantis server. Gitlab pipelines stuck in apply step because this job is in pending state. I saw this feature : #2053 that can explains this behaviour. However, even if this feature is great for GitHub PR workflows, it cannot be used in gitlab.
Do you plan to have a feature flag to deactivate this feature?

[zxpower]

Looks like this is issue not only for Gitlab, but also for Github. We rely on this Github Action: https://github.com/WyriHaximus/github-action-wait-for-status to continue our workflow. And thanks to this "fix" things are now f**ed up. I would propose to include an option to turn on/off this "fix".

[AndreZiviani]

I did the change that broke your workflow, my PR has already been reverted (#2173)
I will try to reimplement it with an option to enable/disable it, sorry for breaking your environment.

[Jonnymcc]

I noticed on our gitlab that after installing Atlantis v0.19.3-pre.20220408 that the apply step is no longer stuck in a "pending" state but now appears "cancelled" and thus the pipeline has a status of "cancelled". Is this expected?

[elisiariocouto]

Is this fixed in 0.19.3?

[nantiferov]

It looks now good in GitHub, no more pending apply.

[syphernl]

On GitLab the general atlantis/apply job remains in pending, even though all individual ones are successful.

[syphernl]

A run on v0.19.4 is "all green" now. No "running" job. I will keep an eye out the coming days to see if this is consistent.
Unfortunately I keep seeing a "running" job.

Can anyone verify this on their Gitlab-based setup?

[vestibuule]

I am still facing the issue on my GitLab environment using Atlantis v0.19.7

[chrislevi007]

We are facing the same issue with atlantis v0.19.7 + Terragrunt and gitlab v14.x

[jamengual]

is this still happening with v0.19.8?

[syphernl]

I believe this is working fine now with v0.19.8. I do not recall seeing pending apply jobs.
Can anyone confirm this?

[syphernl]

Upon further testing this appears to still be an issue.

[chicofranchico]

Same here on 0.19.8 + Gitlab SaaS

[jamengual]

did anyone tested with the latest version? 0.22.2?

[Smixi]

(I removed my comment because I thought it was Atlantis), but a pre-task was executed and lasted for days because the script was waiting for interactive input, and the logs/jobs were not outputted. I couldn't also see the specific jobs logs in Gitlab CI. (Atlantis 19, 19.9 and 21.0)

[DavidPessoa]

I seem to have the same issue on v0.20.0

[nitrocode]

The latest version currently is v0.22.2. I do not think anyone has contributed a fix for this. Please feel free to contribute.

[dcpc007]

Hi we have the same problem since monthes.
Excepting having some pipelines showing running, is there any impact somewhere ? like process/resources usage (and how to see that) ?

[gartemiev]

I didn't find the exact root cause of it but here are some notes from my side:
To by pass this situation I did:

• run: atlantis unlock in PR whiere apply is stuck to remove lockings of this PR

• decline a PR

• create a new branch from old PR and run git commit --amend --allow-empty -m "COMMIT MESSAGE NAME" to change commit hash ID to avoid occuring of old jobs in stuck state in new PR and branch

• next - push changes to this branch and create a PR

A couple notes from my side here:

• It is not recommended to push changes until atlantis finish jobs in scope of existing PR

• when you add a new reviewers to PR - it will cause atlantis triggering again and in case if atlantis jobs for this PR are still running and you added new reviewers - this may also cause this behavior. (I didn’t find a way how to avoid triggering plan during adding new reviewers - so need just be careful and attentive with it).

[lukemassa]

I think this is a similar issue to the one I am in the process of addressing here: #3747.

The issue in this case is this I believe here:

atlantis/server/events/apply_command_runner.go

Line 210 in 79dfa83

status = models.PendingCommitStatus

.

Similar as the issue from #3725, the problem is that we are updating the "atlantis/apply" status (in gitlab a "pipeline") with a pending (in gitlab "running"), despite the step being done. This one is trickier, though, because we can't just exit early, since other VCSs need to be updated as Pending, whereas for gitlab it probably makes the most sense to mark as Success.

I'm only familiar with gitlab so I need to look into this more, but it feels like we can't do "the same thing" for both of them here. Maybe we have to add something to the Client interface that dictates how these Statuses are setup?

[lukemassa]

Looking closer at this, I'm trying to decide whether this is a general problem or a gitlab one.

Particularly, it looks like the current behavior if you plan two projects and only apply one of them is, the apply check says 1/2 and pending.

My question is, do github users think this is a problem or not? I could see either:

1. No it's not a problem. I expect PRs to be in "pending" state when things are still happening on them, and in this case, we've only applied one workspace, so I'd like this functionality to remain
2. Yes it is a problem. I expect PRs to only be in "pending" state when something is actively running, so this is confusing both to users and automation.

I think gitlab is in the 2) camp, since the "Pending" state is used for things that are "running":

Depending on what github users think, I think the fix here would either be to change the logic to say "1/1 applied" and success for everyone, or somehow have a different solution for different VCSs.

[jamengual]

I use github and pending is something that is running, waiting for a status to be reported; this happenes with all github actions.

Pending should not be any longer once the action has been completed.

[lukemassa]

I use github and pending is something that is running, waiting for a status to be reported; this happenes with all github actions.

Pending should not be any longer once the action has been completed.

That's good to know. The good news is that we can and should make this change outside of VCS-specific code, since the expectations seem to be the same. The bad news is, I'm not sure what should be done :)

The best I can come up with is:

1. Two plans are created (plan is 2/2 and green, apply is non-existent)
2. One plan is applied (plan is 2/2 and green, apply is 1/1 and green)

This gets rid of the problem of the "stuck pending", but I do worry about users or automation that say "apply is green" think "this PR is done". @jamengual if this seems like an OK tradeoff to you I can take a stab at writing up the code

UPDATE: Another thing this approach has going for it is, since #3378, if one of the two plans is a no-op, it's going to update apply to be 1/1 anyway, so this would be in line with that

[jamengual]

we need more input on this from other people.

[jamengual]

@nitrocode @GenPage @chenrui333 any inputs?

[william-richard]

I do worry about users or automation that say "apply is green" think "this PR is done".

I worry a lot about this too. I know this would confuse our atlantis users. It's annoying right now to have the status check get in the way sometimes, but I think it would be worse for it to look like everything had been applied, when in reality there are still changes pending that need to be merged.

By marking the apply as successful, when there are still pending plans, I think also will make atlantis consider the code request mergable, while there are still pending changes on it.

[X-Guardian]

Please look at the GitHub/GitLab APIs that are being used, as some of the statements being made here are wrong.

GitLab - Set the pipeline status of a commit. State can be one of: pending, running, success, failed, canceled.
GitHub - Create a commit status. State can be one of: error, failure, pending, success.

[sirpalmeirao]

At this moment i have this problem and a lot of pipelines being stuck on pending.

[jamengual]

I use github and pending is something that is running, waiting for a status to be reported; this happenes with all github actions.
Pending should not be any longer once the action has been completed.

That's good to know. The good news is that we can and should make this change outside of VCS-specific code, since the expectations seem to be the same. The bad news is, I'm not sure what should be done :)

The best I can come up with is:

1. Two plans are created (plan is 2/2 and green, apply is non-existent)
2. One plan is applied (plan is 2/2 and green, apply is 1/1 and green)

This gets rid of the problem of the "stuck pending", but I do worry about users or automation that say "apply is green" think "this PR is done". @jamengual if this seems like an OK tradeoff to you I can take a stab at writing up the code

UPDATE: Another thing this approach has going for it is, since #3378, if one of the two plans is a no-op, it's going to update apply to be 1/1 anyway, so this would be in line with that

I think pending is something in an "uncompleted state," so based on your recommendation, I think this will clarify the definition of a pending state, so I agree with that approach.

If a user plans two projects and later applies only one, then that is their decision to do so, and if the apply is successful, then it should go green.

The status in itself can't figure out what the user is doing with the Atlantis commands.

[brontolinux]

I am seeing this with Atlantis 0.28.5 (cannot upgrade because of #4950), Gitlab 17.3 and 17.4, and a custom workflow to use terragrunt run-all, adapted from the documented custom workflow. The atlantis.yaml file is generated via terragrunt-atlantis-config.

When the MR changes more than one project, plan works fine. With apply, the atlantis/apply pipeline remains running, and apply itself is run on one project, or none. The error in the logs is the well known: Cannot transition status via :run from :running.

applying the plans one by one works on 17.3; I haven't tested in 17.4 but assuming it still works. Update: it works on 17.4, too. After the last plan is applied, the whole pipeline appears as completed.