[enhancement]: Support Sparse Checkout

[bdovaz]

Describe your feature request here

For monorepo repositories this is a must to be able to speed up builds as much as possible.

I see that this very thing has been recently merged by @dscho into the GitHub checkout action: actions/checkout#1369

Would this same functionality be possible in azure pipelines?

Thanks!

[KonstantinTyukalov]

Hi @bdovaz thanks for the reporting! We're currently working on higher priority issues, but we'll get back this this one soon.

[dscho]

Note that the actions/checkout PR relies on partial clones to be supported server-side, something that Azure Repos could only do via emulation using the GVFS protocol (which so far is only supported using Microsoft's Git fork, which is not installed on the build agents by default).

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[bdovaz]

No stale

[dscho]

Note that the actions/checkout PR relies on partial clones to be supported server-side, something that Azure Repos could only do via emulation using the GVFS protocol (which so far is only supported using Microsoft's Git fork, which is not installed on the build agents by default).

Note that partial clones are now supported by Azure Repos!

[bdovaz]

Although it is an improvement, it has nothing to do with the purpose of this issue.

The big improvement would be that at a high level through the checkout task we can make a sparse checkout indicating the directories to include as you can do in Github from your PR from half a year ago.

[sanghel-orbyta]

any way to bump priority on this one?

monorepos consume agent time uselessly doing full checkouts...

[ToreDemant]

Seems to be related to these:
https://developercommunity.visualstudio.com/t/Sparse-Checkout--Partial-Clone-for-nati/10607236
https://developercommunity.visualstudio.com/t/Support-Gits-sparse-checkout-in-the-Azu/10472337

[Khayoann2]

I proposed this one
https://developercommunity.visualstudio.com/t/Sparse-Checkout--Partial-Clone-for-nati/10607236
So i’m also waiting this feature …
Please 🙏

[mggrand]

If anyone is interested, I wrote the sparse checkout as a template, which can then be used as follows:

- template: sparse-checkout.yml
    parameters:
      folders:
        - frontend
        - resources

with the template sparse-checkout.yml

parameters:
  - name: folders
    type: object
    default:

  - name: fetchLfs
    type: boolean
    default: true

  - name: persistCredentials
    type: boolean
    default: false

steps:
  - checkout: none
  - pwsh: |
      function Start-Group($name){ Write-Host "##[group]$name"}
      function End-Group { Write-Host "##[endgroup]"}

      function Run-Cmd($command){
          Write-Host "##[command]$command"
          Invoke-Expression $command
      }


      $folders = "${{ join(' ',parameters.folders) }}"
      $repositoryUri = "$(Build.Repository.Uri)"
      Write-Host "Sync repository folders '$folders' to '$pwd'"
            
      Start-Group "Getting git version info"
      Run-Cmd "git version"
      Run-Cmd "git lfs version"
      End-Group

      if(Test-Path ".git"){
        Start-Group "Cleanup existing repository"
        # Existing directory
        $null = Remove-Item ".git/index.lock" -Force -ErrorAction SilentlyContinue
        $null = Remove-Item ".git/shallow.lock" -Force -ErrorAction SilentlyContinue

        $currentRemote = Run-Cmd "git remote get-url origin"
        if($currentRemote -ne $repositoryUri){
            Write-Host "Current fetch url: $currentRemote"
            Write-Host "New repository fetch url: $repositoryUri"
            Write-Host "##[warning]Fetch urls do not match. Clean the whole directory"
            $null = Remove-Item ./* -Force -Recurse
        }
        else{
            Run-Cmd "git remote remove origin"
            Run-Cmd "git clean -ffxd"
            Run-Cmd "git reset --hard"
        }
        End-Group
      }

      Start-Group "Initialize the repository"
      Run-Cmd "git init"
      Run-Cmd "git sparse-checkout init --cone"
      if($folders.Length -gt 1){
        Run-Cmd "git sparse-checkout add $folders"
      } else {
        Write-Host "No folders were provided for the sparse checkout"
      }
      Run-Cmd "git lfs install --local"
      End-Group

      Start-Group "Configure the repository"
      Run-Cmd "git remote add origin $repositoryUri"
      Run-Cmd "git config core.sparsecheckout true"
      Run-Cmd "git config gc.auto 0"
      Run-Cmd "git config --get-all http.$($repositoryUri).extraheader"
      Run-Cmd "git config --get-all http.proxy"
      Run-Cmd "git config http.version HTTP/1.1"
      End-Group


      Start-Group "Fetching the repository"
      Run-Cmd "git --config-env=http.extraheader=env_var_http_extraheader fetch --force --no-tags --progress --prune --no-recurse-submodules --verbose --depth=1 origin +$(Build.SourceVersion):refs/remotes/origin/$(Build.SourceVersion)"
      if("${{ parameters.fetchLfs }}" -eq "true"){
        Run-Cmd "git --config-env=http.extraheader=env_var_http_extraheader lfs fetch origin refs/remotes/origin/$(Build.SourceVersion)"
      } else {
        Write-Host "Skipping lfs fetch"
      }
      End-Group


      Start-Group "Checkout the repository"
      Run-Cmd "git checkout --force --progress refs/remotes/origin/$(Build.SourceVersion)"
      End-Group

      if("${{ parameters.persistCredentials }}" -eq "true"){
        Start-Group "Persist credentials in the repository"
        Run-Cmd "git config http.$($repositoryUri).extraheader `"$($env:env_var_http_extraheader)`""
        End-Group
      }

      if("$($env:env_var_vsts_debug)" -eq "true"){ 
        Start-Group "Print directory outputs"
        Run-Cmd "Get-ChildItem -Recurse"
        End-Group
      }

    displayName: "Clone partial git repo"
    env:
      env_var_http_extraheader: "AUTHORIZATION: bearer $(System.AccessToken)"
      env_var_vsts_debug: $[eq(variables['System.Debug'], true)]