Sparse checkout for git pulls

[tetracionist]

Added support for sparse-checkout when cloning a GitHub repo (#15185).
This is ideal for teams that have larger repos who don't want to clone all the folders in their repo.

When directories are specified in the Prefect.yaml or GitHubRepository storage class, we will use sparse checkout to get the directories and any file in the root directory (cone-mode).

Example usage in a Prefect.yaml

pull:
- prefect.deployments.steps.git_clone:
    repository: https://github.com/tetracionist/prefect.git
    branch: sparse-checkout-for-git-pulls
    access_token:
    directories: [src/integrations/prefect-azure, src/integrations/prefect-dask]

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

CodSpeed Performance Report

Merging #15824 will not alter performance

_{Comparing tetracionist:sparse-checkout-for-git-pulls (2d64115) with main (40590a5)}

Summary

✅ 3 untouched benchmarks

[desertaxle]

Thanks for the PR @tetracionist! This is looking pretty good, but I have a couple of questions about this implementation after a first pass.

[desertaxle]

It looks like the init subcommand has been deprecated, and the set subcommand is the preferred approach based on the git docs. It seems like we would update this method to only call git sparse-checkout set with the provided directories, but let me know if that would cause issues.

[tetracionist]

I think this should be okay, I will try using git sparse-checkout set

[desertaxle]

It looks like if sparse checkout mode is used then you won't be able to include submodules in the checkout. Is is possible to have both?

[tetracionist]

I put this in the else block as I was unsure what submodules did, but are they something to do with adding code from another repo into the cloned repo?

I think this would be easy with cone mode but might be trickier without.
Would it be useful to also sparse-checkout the submodule?

[desertaxle]

Yeah, submodules allow you to reference another git repository from a parent repository. I'm not sure how sparse checkout would interact with submodules, but I think it'd be useful to apply sparse checkout to submodules, also.

[github-actions]

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.

[github-actions]

This pull request is stale because it has been open 14 days with no activity. To keep this pull request open remove stale label or comment.

[tetracionist]

@desertaxle Thanks for reviewing back in October, I had a chance to have another look at this PR again.

The main changes I have made are as follows:

• Now using sparse-checkout set instead of init
• Added an async function to check if repository is in sparse-checkout mode
• Removed cone-mode
• Two new tests for sparse-checkout
• Can still use submodules if using sparse-checkout
• I ditched the filter blob none and ended up sticking with shallow clones (after reading these docs it seemed to be the most sensible option)

I also used the configuration and flow below to test what folders get cloned and that worked as expected (brings back all items in orchestration and ingestion folder, and anything in the root directory).
Please would you be able to to review this again?

definitions:
  actions:
    pull_staging: &pull_staging
      - prefect.deployments.steps.git_clone:
          repository: https://github.com/tetracionist/repo.git
          branch: test-sparse
          access_token: '{{ prefect.blocks.secret.my-gh-token }}'
          directories: ["orchestration", "ingestion"]

from pathlib import Path

from prefect import flow
from prefect.logging import get_run_logger


@flow(name="dir check flow")
def dir_check_flow():

    logger = get_run_logger()
   
    directory = Path()
    logger.info(f"{list(directory.rglob("*"))}")


if __name__ == "__main__":
    dir_check_flow()