Can't get .git directory to show up in remote builds

[abevoelker]

I'm submitting a remote build with gcloud container builds submit --config cloudbuild.yaml and no matter what I try, the .git directory is not showing up in the temporary tarballs. My Dockerfile needs to access the .git directory for the build and obviously it can't find it and all my remote builds fail (local builds work fine).

There is no .dockerignore, no .gitignore, no .gcloudignore, and no "Some files were not included in the source upload." message like I get in other projects where I use those ignore files.

If I create a different hidden directory:

mkdir .foo
touch .foo/bar

that directory does show up in the tarball, so I know it's not excluding all hidden directories.

Is there some hardcoding specifically to exclude .git when creating the tarball?

[bendory]

Indeed, if you look at the code for gcloudignore, you'll see that we generally ignore a .git directory, since it is rather unusual to want to upload this as part of your build context.

If you want it, you have several options:

• create an empty .gcloudignore (touch .gcloudignore)
• create a .gcloudignore (or .gitignore) that ignores the files you don't want but doesn't ignore .git
• create a .gcloudignore that explicitly includes .git (echo '!.git' > .gcloudignore)
• turn off the ignore behavior completely: gcloud config set gcloudignore/enabled false

See the gcloudignore documentation for further details.

[abevoelker]

Thank you for the detailed reply, I got it working with your suggestions. 🎈 🎉

I agree it is rather unusual (I used to always add .git to .dockerignore), but I like making the git revision available to my web app for use in a certain admin panel (after that part of the build .git is deleted), e.g.

RUN \
  git rev-parse --verify HEAD > /var/www/.git-revision &&\
  rm -rf /var/www/.git

[grantstephens]

Cloud this please be documented somewhere. I can't find any reference to default .gcloudignore items in the docs

[bill-within]

@bendory is there a different logic applied when using the GCB GitHub App? I'm trying to get private submodules working, but the extracted tarball doesn't contain a .git folder

[bendory]

redirect to @wlynch who will know if there's a different logic in the app

[tjanson]

I think it’s worth reopening this issue. I appreciate the thorough answer, @bendory, but it seems the .gcloudignore isn’t used when builds are triggered from GitHub.

Or is there a separate issue tracking that in particular?

[JohnPaton]

I'm also running into this: with the .git directory is not being added to my image in Cloud Build triggered by GitHub, despite having an empty .gcloudignore present.

[saminiemi]

If found it will work with GitHub this way:

• First connect your GitHub repository to Cloud Source Repositories (https://https://source.cloud.google.com/)
• Add GitHub repository as an external repository
• Create Build Trigger agains mirrored repository

In this way there is .git folder available in workdir during the build process (still need include it on .gcloudignore)

[JohnPaton]

The following build step fetches the .git directory as long as your build has access to your repo:

  - id: Get the .git directory
    name: git
    args:
      - bash
      - -c
      - |-
        # get the .git directory only
        git clone --branch $BRANCH_NAME git@github.com:<USER_NAME>/$REPO_NAME /tmp/repo ;
        # move it to the workspace
        mv /tmp/repo/.git .

[mr-smithers-excellent]

Also experiencing this issue with the new Cloud Build GitHub App triggers. Have attempted to explicitly add !.git to the .gcloudignore file with no success.

@bendory @wlynch - have you been able to recreate this on your end?

[redging-very-well]

@saminiemi Your workaround doesn't work if you want to take advantage of the "Build on PR" feature that's only available with the Cloud Build GitHub App repo :(

[tlbdk]

We do this instead of moving the repo:

  # checkout git submodules
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
    - '-c'
    - |
      git init
      git remote add origin git@github.com:/orgname/$REPO_NAME.git
      git fetch --depth=1 origin $COMMIT_SHA
      git reset --hard FETCH_HEAD
      git config -f .gitmodules submodule.core.url git@github.com:/orgname/$REPO_NAME.git
      git submodule update --init

And for authentication we use this project:
https://github.com/connectedcars/auth-wrapper

[Datamance]

Hi all,

There's a much quicker (still hacky) way to get the .git folder in without having to download the entire world all over again:

- id: get-git-history
  name: gcr.io/cloud-builders/git
  entrypoint: bash
  args:
    - '-c'
    - |
      git clone https://github.com/lookit/lookit-api.git --no-checkout
      mv lookit-api/.git .
      git fetch --unshallow
      rm -rf lookit-api

Basically, use --no-checkout to only get the .git folder, then move it up one level, then delete the unnecessary checkout folder.

[matthiasa4]

Another alternative is using Google Cloud Secret Manager - put your GitHub OAuth token and retrieve in the Cloud Build image:

Adding secret:

echo $GITHUB_TOKEN | gcloud secrets create github-token --data-file=- --replication-policy=automatic

Retrieving secret in Build pipeline:

GITHUB_TOKEN=$(gcloud secrets versions access latest --secret=$GITHUB_SECRET_NAME)
COMMAND=$(echo 'git clone https://GITHUB_TOKEN@github.com/ORGANISATION/REPOSITORY' | sed -e "s/GITHUB_TOKEN/$GITHUB_TOKEN/" )
eval $COMMAND

[mrmeku]

Here's a solution that you can use without having to authenticate with Github:

First connect your GitHub repository to Cloud Source Repositories (https://https://source.cloud.google.com/)

Then use the gcloud builder to auth against your cloud source repo:

- name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
      - git init
      - git config credential.helper "!gcloud auth git-helper --account=<SERVICE-ACCOUNT-EMAIL> --ignore-unknown $@"
      - git remote add origin https://source.developers.google.com/p/<CLOUD_SOURCE_REPOSITORIES_PATH>

Note that the value for SERVICE-ACCOUNT-EMAIL can be found under https://console.cloud.google.com/cloud-build/settings/service-account

Also note that after you are no longer in the gcr.io/cloud-builders/gcloud container you will no longer be able to use git commands since gcloud will no longer be installed/correctly configured in other docker images.

Good luck everyone!

[JohnPaton]

Really the solution here is just for the Cloud Build app to send .git/ with the rest of the files when a build is triggered, it's clearly something that people need and its unintuitive that only this one directory is missing.

Where would be the right place to file an issue about this?

[realpha]

Heavily needed! Gitops without a rev history is like trying to eat soup with a fork...

[realpha]

If you need access to the full repo, the simplest solution that I've found is the following:

steps:
  - name: gcr.io/cloud-builders/git
    args: ["fetch", "--unshallow", "--tags"]

No need to move the .git directory around, as it converts your shallow clone into a full copy.
Alternatively to fetching your whole repo, you could also specify --depth=<NUMBER>.

Credit: This guy's company blog

[Datamance]

@realpha I think that only works with GSC repos, no?

[JohnPaton]

I think for a private GitHub repo this will leave you with some extra hoops to jump through still, right?

[realpha]

@Datamance @JohnPaton Sorry, I forgot to mention that you have to mirror your repository, otherwise it won't work.
I have to admit, that this won't be suitable solution in every situation. For our workflow (Sanity and regression test at PR, full tests and build upon merge) it works fine.

[noelbk]

Why is this closed? Having a .git directory is still required for github triggered builds.

we generally ignore a .git directory, since it is rather unusual to want to upload this as part of your build context.

That assumption is false. We need the .git directory for several reasons, including git describe, git submodule, and git lfs

Here's a cloudbuild.yaml file I made to hack around this. This checks out the .git directory and checks out a private submodule with git-lfs files.

steps:
- name: 'ubuntu+git-lfs'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    set -eux
    
    # cloudbuild doesn't check out the .git directory, so I have to
    # check it out again to get the .git directory, submodules, and
    # LFS files
    eval $(ssh-agent -s)
    mkdir -p -m 0700 /root/.ssh
    cat ci/host-git.luolix.top >> /root/.ssh/known_hosts

    # MYREPO1.key was made from ssh-keygen then added to the github
    # repo's Deploy Keys https://github.com/USER/REPO/settings/keys
    chmod 0600 ci/MYREPO1.key
    ssh-add ci/MYREPO1.key
    git init
    git lfs install
    git remote add origin git@github.com:/GITHUB_USER/$REPO_NAME.git
    git fetch --depth=1 origin $COMMIT_SHA
    git reset --hard FETCH_HEAD
    # have to switch to MYSUBREPO2.key to check out the MYSUBREPO2 submodule
    ssh-add -D
    chmod 0600 ci/MYSUBREPO2.key
    ssh-add ci/MYSUBREPO2.key
    # if there were other private repos here, I'd have to switch keys
    # and update them individually
    git submodule update --init --recursive
    # finally the .git repo is rebuild and checked out
        
    make test

[JohnPaton]

Retry: pinging @wlynch (and @bendory to reopen the issue)

[Stono]

Just stumbled across this, it's beyond silly! Cloud Build is forcing a gitops model, and you don't have access to git 🤯

[seenkey]

As noted above, the presence of the .git directory depends on how the build was created, and that directory is indeed not currently available when using the GitHub App. We're aware of the issue, please stay tuned.

This issue remains closed as this isn't something that we plan to address within cloud-builders, even as there are various workarounds listed above which may work for some people/workflows.

[noelbk]

that directory is indeed not currently available when using the GitHub App

Interesting. That's not a problem using other GitHub apps like Travis or CircleCI. gcloud has a read-only SSH key to fetch the whole repo, so why doesn't gcloud check out the whole branch with git fetch --depth=1 origin $COMMIT_SHA ?

Also, please don't forget to add git-lfs to fetch LFS files.

[wlynch]

Hi! Responding to the ping. I no longer work on the GCB GitHub App, but I can provide some context about why things are the way they are.

The previous GitHub integration mirrors the repo to Cloud Source Repositories, and performs a git clone on the CSR copy. While this simplifies auth on the GCP side (e.g. you can clone using the gcloud helper), this is prone to sync issues.

One of the goals of the GitHub App were to resolve some of these pain points by fetching repos directly from GitHub. However, when the GCB App was first launched there was no mechanism to scope installation tokens, so a GCP project could hypothetically affect any repo in the installation since you can only have 1 installation per owner (e.g. user or org).
This was a problem since this wasn't suitable for orgs that had multiple repos shared across GCP projects (e.g. https://github.com/google are supported by many teams, each with their own GCP project), since one installation token could get access to anything in the installation and perform any action the App was granted. Because of this, we decided to not expose the installation token to GCB workers and stage the source in a read-only location to avoid this issue, without going as far as to maintain complete mirrors of the repo again (we also couldn't generate read-only SSH keys without requesting repo admin permissions, which we wanted to avoid).

Since then, GitHub has added the ability to scope installation tokens by repo and permission, which looks promising to address the problems above. That said, there's probably more to think through and there's always priorities to juggle. ;) As @koolte mentioned, the team is aware of this.

In general, https://issuetracker.google.com/issues/new?component=190802&template=1162663 tends to be the best mechanism to raise a bug or show interest in a particular feature.

In the meantime, as others have mentioned, as a workaround you can perform git operations yourself in your builds. If you require auth, you can provide credentials to your worker via https://cloud.google.com/cloud-build/docs/securing-builds/use-encrypted-secrets-credentials.

While this may not be the update you are hoping for, I hope this helps explain how things came to be!

[noelbk]

Thanks a lot for the detailed reply, @wlynch ! Hopefully this can be fixed now that installation tokens can be scoped. For now, I have a hack to check out the rest of my repo, submodules, and LFS files with my own token. It works.

Thanks for the link to the cloudbuild build issue form for future issues

[seenkey]

See https://issuetracker.google.com/issues/136435027