Repo-server doesn't fully clean workspace

[mrysavy]

Describe the bug

When I have two Applications with same git repo (but different branch) there is a situation, when repo-server doesn't fully clean working directory of cloned repo. Directories containing git working directory (including .git subdir) are not cleaned when changing branch by repo-server.
The specific situation is when the one branch has git submodule, but the other one doesn't. After switching the branch, the directory with submodule are kept, even when the new branch doesn't contain it.
It is applicable not only to submodule, but to any subdir with regular .git folder. And it is not applicable only when switching branch, but in the case of hard refresh too.

This is a big problem when the git submodule is subchart because after switching to another branch (branch without that subchart) the subchart is also applied and the deployment doesn't correspond to git.

I've checked that repo-server uses "git clean -fdx", when (according to doc) single "-f" preserves untracked nested git repositories. Using double force "git clean -ffdx" solves the issue.
But better solution may be using "--recurse-submodules" git-checkout argument (as stated here: https://git-scm.com/book/en/v2/Git-Tools-Submodules#Issues-with-Submodules).

To Reproduce

1. Using argocd-repo-server pod terminal create subdir in the repo working dir
2. Copy any regular .git folder (including content) to that subdir
3. Create another empty subdir (for comparing)
4. Do application Hard refresh in argo web UI
5. Check the repo working dir on repo-server pod
   • subdir contain .git folder remained
   • empty folder is gone

Expected behavior

Full working dir cleanup in all situations when the cleanup is in place.

Version

argocd: v1.5.5+0fdef48
  BuildDate: 2020-05-16T04:02:57Z
  GitCommit: 0fdef4861e12026e133224f7c9413072340e2983
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.5.5+0fdef48
  BuildDate: 2020-05-16T04:04:43Z
  GitCommit: 0fdef4861e12026e133224f7c9413072340e2983
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-11T03:12:59Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

And I have one question: Is it possible to have .gitconfig with "submodule.recurse true" as a workaround?

[domeales-paloit]

I think I have run into this now when updating my submodule to use https over ssh - the repo server didn't seem to pick up the update. It looks like this major bug is not yet resolved..

[yann-soubeyrand]

I confirm that this issue is still present in version 2.6.