Unable to merge pull (GitHub App) - Atlantis auto merge fails once project is successfully applied

[kpettijohn]

We recently noticed our Atlantis server is unable to perform an auto merge on a successful apply. Perviously auto merge was working as expected but recently (April 26th) it stopped working for us.

Here are a few details on our environment but please let me know if any additional details would be helpful.

• Atlantis version 0.21.0
  • We also upgraded to 0.23.5 and experienced the same issue.
• We use Github with the Github App permissions outlined here: https://www.runatlantis.io/docs/access-credentials.html#github-app
• We enable automerge: true within each repos atlantis.yaml file.
• More recently we stated using Github org rulesets but these were enabled about 10 days before we noticed the auto merge failures . I still need to disable the org level rulesets entirely to see if that fixes our auto merge issue.
  • Also just to note but when looking at the org repo rule insights I don't see any failures for our Atlantis Github app.

Error message from when the auto merge is attempted by Atlantis.

{
  "level": "error",
  "ts": "2023-05-04T14:33:08.040Z",
  "caller": "vcs/instrumented_client.go:231",
  "msg": "Unable to merge pull, error: merging pull request: PUT https://api.github.com/repos/<org_redacted>/<repo_redacted>/pulls/77/merge: 500  []",
  "json": {
    "pull-num": 77
  },
  "stacktrace": "github.com/runatlantis/atlantis/server/events/vcs.(*InstrumentedClient).MergePull\n\tgit.luolix.top/runatlantis/atlantis/server/events/vcs/instrumented_client.go:231\ngit.luolix.top/runatlantis/atlantis/server/events/vcs.(*ClientProxy).MergePull\n\tgit.luolix.top/runatlantis/atlantis/server/events/vcs/proxy.go:80\ngit.luolix.top/runatlantis/atlantis/server/events.(*AutoMerger).automerge\n\tgit.luolix.top/runatlantis/atlantis/server/events/automerger.go:35\ngit.luolix.top/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgit.luolix.top/runatlantis/atlantis/server/events/apply_command_runner.go:166\ngit.luolix.top/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgit.luolix.top/runatlantis/atlantis/server/events/command_runner.go:296"
}

Overall I'm hoping folks might be able to provide some suggestions on what our problem might be. I have looked for similar auto merge issues but the ones I found seem to provide a more specific error message and took place after an upgrade (#3268).

[jamengual]

did atlantis user/app permissions change? maybe does not have permission to merge?

[kpettijohn]

Interesting... thanks for the heads up. Unfortunately we have been seeing this issue consistently now for weeks on any automerge attempted by Atlantis so it doesn't seem related to Github having issues but they do seem to be having issues pretty frequently more recently 😬

[jamengual]

if you did not change anything in your infra or atlantis setup then you need to look at if you might be hitting any API limits in github, AFAIK github apps have lower limits than webhooks.

[kpettijohn]

Rate limits all looked good. It turns out this issue is somehow related to our Github repo rulesets we have enabled on our org. After excluding all of our org rule sets from a specific repository the Atlantis auto merge worked successfully. Unfortunately Github org "rule insights" didn't seem to provide any details on why the merge request failed. I need to dig into things more to see if I can find the exact rule that was causing the issue.

Thanks for taking the time to provide some suggestions!

[jamengual]

please let us know what you find

[kpettijohn]

Another team member spent time looking into this issue and it seems it might be caused when "Bypass Mode" is set to "Organization bypass permitted" on some of our rules. Overall the behavior is a bit odd so we are attempting to confirm if this is the expected behavior or potentially a bug with Repository Rules.

More details in the following discussion: https://github.com/orgs/community/discussions/52652#discussioncomment-5863010