Atlantis Doesn't Run on Google Cloud Run

[gaahrdner]

I can't seem to get Atlantis to run on Google's Cloud Run. I think this could be a limitation of the runtime sandbox but I'm not sure. The Atlantis documents do say that there is no permanent storage so I'm concerned I might be missing something simple. The command I'm passing (just for testing right now) is:

          "server",
          "--bitbucket-user=FOO",
          "--bitbucket-token=BAR",
          "--repo-whitelist=bitbucket.org/organization/*"

Unfortunately, I receive the following error:

"Application failed to start: Failed to create init process: Failed to load server --bitbucket-user=FOO --bitbucket-token=BAR --repo-whitelist=bitbucket.org/organization/*: no such file or directory"

I thought the usage of dumb-init from the base image would have alleviated this?

[lkysow]

Those docs are a bit misleading. You need to store your Terraform state somewhere because after the PR is merged Atlantis deletes the data for that PR. However in between the PR opening and you running atlantis apply, Atlantis needs to store the plan and other data. Thus you can't run on Cloud Run unless there's a way to mount in a persistent disk.

[MPV]

Related: #894 (external backend for locking)

[lkysow]

I'm going to close this since in order for it to work we'd need #894 and other work done so this ticket is a duplicate. If you're interested in this, subscribe to those tickets.

[ademariag]

Is this the case even now that you can mount one of these filesystems? https://cloud.google.com/run/docs/using-network-file-systems

[IchordeDionysos]

Cloud Run should now be possible even without any changes to how Atlantis works:
gaahrdner/atlantis-on-cloud-run#1 (comment)

[gaahrdner]

Did you get this working @IchordeDionysos ?

[IchordeDionysos]

Unfortunately, I haven't had the time to investigate whether it works or not ...
Maybe I'll find the time sometime soon?!

[IchordeDionysos]

Okay, I have made some more progress toward this ☺️

This is what I'm working with right now:

Dockerfile

FROM ghcr.io/runatlantis/atlantis:v0.27.1-alpine

USER root

# Not sure if this is needed?! I think it was for installing tflint
RUN apk add dpkg

# Install tflint
# RUN curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | bash

# Install tfsec
# RUN curl -s https://raw.githubusercontent.com/aquasecurity/tfsec/master/scripts/install_linux.sh | bash

# Set fallback mount directory
ENV MNT_DIR /app/atlantis

WORKDIR  $MNT_DIR

# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY repos.yaml ./

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["server"]

Cloud Run service.yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: atlantis
  namespace: <project-id>
  labels:
    cloud.googleapis.com/location:<region>
  annotations:
    run.googleapis.com/launch-stage: BETA
    run.googleapis.com/description: "Atlantis is a tool for collaborating on Terraform through GitHub."
    run.googleapis.com/ingress: all
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "1" # Forces an instance to always be active
        autoscaling.knative.dev/maxScale: "1" # Maybe those could be tweaked and more instances spun up given it's all persisted in a central location?
        run.googleapis.com/cpu-throttling: "false" # Always allocate CPU as I think Atlantis does some operations in the background?! Or at least expects to run operations in the background
        run.googleapis.com/startup-cpu-boost: "true"
        run.googleapis.com/execution-environment: "gen2" # This was needed for my previous attempt given it's a more complete server environment
    spec:
      containerConcurrency: 80
      timeoutSeconds: 300 # todo: Might need to be increased for complex projects
      serviceAccountName: <service-account>
      containers:
        - name: atlantis
          image: <region>-docker.pkg.dev/<project-id>/<path>/atlantis@<sha-hash>
          ports:
            - name: http1
              containerPort: 8080
          env:
            - name: ATLANTIS_ATLANTIS_URL
              value: <atlantis-url, e.g. Cloud Run URL>
            - name: ATLANTIS_GH_TOKEN
              value: <github-token>
            - name: ATLANTIS_GH_USER
              value: <bot-user>
            - name: ATLANTIS_GH_WEBHOOK_SECRET
              value: <webhook-secret>
            - name: ATLANTIS_PORT
              value: '8080'
            - name: ATLANTIS_REPO_CONFIG
              value: /app/repos.yaml
            # ... other ATLANTIS config not yet present in repos.yaml
            - name: ATLANTIS_DATA_DIR
              value: /app/atlantis
          resources:
            limits:
              cpu: 1000m
              memory: 2Gi
          volumeMounts:
            - name: atlantis_volume
              mountPath: /app/atlantis
          startupProbe:
            timeoutSeconds: 240
            periodSeconds: 240
            failureThreshold: 1
            tcpSocket:
              port: 8080
      volumes:
        - name: atlantis_volume
          csi:
            driver: gcsfuse.run.googleapis.com
            readOnly: false
            volumeAttributes:
              bucketName: your-atlantis-state-bucket

Previously I had issues with getting git to work in the first place, now I'm able to start cloning the repo into Cloud Storage, but it fails intermittently when creating folders ...

Example output

running git clone --depth=1 --branch <branch> --single-branch https://<gh-bot-user>:<redacted>@github.com/simpleclub/infrastructure.git /app/atlantis/repos/<org>/<repo>/259/default: Cloning into '/app/atlantis/repos/<org>/<repo>/259/default'...
Updating files:   2% (7/252)
Updating files:   3% (8/252)
Updating files:   3% (9/252)
Updating files:   4% (11/252)
Updating files:   5% (13/252)
Updating files:   5% (15/252)
Updating files:   6% (16/252)
Updating files:   7% (18/252)
Updating files:   8% (21/252)
Updating files:   9% (23/252)
fatal: cannot create directory at 'modules/cloud_build/cloud_build_deployment': Interrupted system call
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

: exit status 128

If someone has an idea how to improve it, would be happy to hear it ☺️

[IchordeDionysos]

There is an open issue which seems to be actively developed by Google: GoogleCloudPlatform/gcsfuse#1016
Once that's fixed I'm hopeful Atlantis on Cloud Run would be as simple as building a docker image, pushing to Artifact Registry and deploying Cloud Run.

[gaahrdner]

Thanks a lot for tracking this down @IchordeDionysos

[IchordeDionysos]

Okay, there has been some progress on the gcsfuse side.
They have merge a fix (behind a command line flag), I've just tested it and it did work 🙌
I was able to successfully run plan a plan!

For me the next steps would be, to wait until the next version of gcsfuse is deployed (27th May) and picked up by Cloud Run.
This will hopefully happen sometime in June.

Then I would prepare a full example that we can use to deploy Atlantis on Cloud Run.

[blastdan]

Hey @IchordeDionysos So it looks like the Gcsfuse v2.3.0 supports the required feature you needed out of the box.

Would you be able to show the example that you are using for running Atlantis on CloudRun?

Thanks!

[IchordeDionysos]

I'm currently quite busy, but I wanted to give it some time so that Cloud Run hopefully uses Gcsfuse 2.3.0. It's not really visible which version they are using.

I might get to it in the next week(s).

[IchordeDionysos]

I've just tested it, and Cloud Run still uses the old version of GCS Fuse...

[m0ps]

Good news...
Very soon, gcpfuse mount options will be available for Cloud Run, so a dream come true.
eg:

mountOptions: uid=100,gid=1000,file-mode=0744,only-dir=atlantis-data

[m0ps]

But there is another problem... /healthz endpoint is not available because of the following Cloud Run limitation:

Some paths ending with z. We recommend avoiding all paths that end in z to not collide with reserved paths.

https://cloud.google.com/run/docs/known-issues#ah

[valorl]

Good news... Very soon, gcpfuse mount options will be available for Cloud Run, so a dream come true. eg:

mountOptions: uid=100,gid=1000,file-mode=0744,only-dir=atlantis-data

@m0ps Could you please share the source for this information? Wasn't able to find it easily

[IchordeDionysos]

Actually I think I've managed to get it to work now 🥹

Let me write up some guide on this later today ☺️

[IchordeDionysos]

I set up a quick repository to collect examples to help people start using Atlantis on Cloud Run:
https://github.com/simpleclub-extended/atlantis-cloud-run

As this is quite a different way to run Atlantis, and there might be unique challenges, I hope that we can collect some knowledge and experience on what works best running Atlantis on Cloud Run.

Feel free to play around with different configurations, improve existing or add other examples and share your experiences on what works best.

[m0ps]

@valorl
Now it's official:
https://cloud.google.com/run/docs/configuring/services/cloud-storage-volume-mounts#volume-mount-options