Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial sparse implementation #54

Merged
merged 6 commits into from
Aug 22, 2019
Merged

Initial sparse implementation #54

merged 6 commits into from
Aug 22, 2019

Conversation

derrickstolee
Copy link
Contributor

@derrickstolee derrickstolee commented Aug 14, 2019 •
edited
Loading

Uses the code currently at microsoft/git#180. Covers the basics of #8.

  • The default scalar clone runs git sparse-checkout init so the working directory only has files at root.

  • Run git sparse-checkout add <folders.txt to pipe in a list of folders, and it will expand those files.

Example workflow:

$ scalar clone https://dev.azure.com/gvfs/ci/_git/ForTests
Clone parameters:
  Repo URL:     https://dev.azure.com/gvfs/ci/_git/ForTests
  Branch:       Default
  Cache Server: Default
  Local Cache:  C:\.scalarCache
  Destination:  C:\_git\test2\ForTests
Authenticating...Succeeded
Querying remote for config...Succeeded
Using cache server: None (https://dev.azure.com/gvfs/ci/_git/ForTests)

WARNING: Unable to validate your Scalar version
Server not configured to provide supported Scalar versions

Cloning...Succeeded
Fetching commits and trees from origin (no cache server)...Succeeded
Validating repo...Succeeded
Mounting...Succeeded

$ cd ForTests/src/
$ ls
AuthoringTests.md  GvFlt_EULA.md  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ echo GVFS/GVFS.Common >>../folders.txt
$ echo GVFS/GVFS.UnitTests >>../folders.txt
$ echo GitHooksLoader >>../folders.txt
$ git sparse-checkout add <../folders.txt

$ ls
AuthoringTests.md  GitHooksLoader/  GvFlt_EULA.md  GVFS/  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ echo GVFS/GVFS >>../folders2.txt
$ echo GVFS/GVFS.FunctionalTests >>../folders2.txt
$ git sparse-checkout add <../folders2.txt

$ ls
AuthoringTests.md  GitHooksLoader/  GvFlt_EULA.md  GVFS/  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ ls GVFS
GVFS/  GVFS.Common/  GVFS.FunctionalTests/  GVFS.UnitTests/  LibGit2Sharp.NativeBinaries.props  ProjectedFSLib.NativeBinaries.props

$ git sparse-checkout list
/
/GVFS/
/GVFS/GVFS/*
/GVFS/GVFS.Common/*
/GVFS/GVFS.FunctionalTests/*
/GVFS/GVFS.UnitTests/*
/GitHooksLoader/*

I spun up a few remaining issues for follow-up work: #76, #77, #78.

@derrickstolee derrickstolee mentioned this pull request Aug 16, 2019
Closed
@github-actions
Copy link

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@github-actions github-actions bot added the no-activity Stale issue or PR that will be automatically closed soon. label Aug 16, 2019
@derrickstolee
Copy link
Contributor Author

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@jrbriggs is this a bug in the YAML, or a bug in Actions after enabling the task?

@derrickstolee derrickstolee added performance and removed no-activity Stale issue or PR that will be automatically closed soon. labels Aug 16, 2019
@github-actions
Copy link

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@github-actions github-actions bot added the no-activity Stale issue or PR that will be automatically closed soon. label Aug 17, 2019
@jrbriggs jrbriggs removed the no-activity Stale issue or PR that will be automatically closed soon. label Aug 17, 2019
@jrbriggs
Copy link
Member

| @jrbriggs is this a bug in the YAML, or a bug in Actions after enabling the task?

@derrickstolee Happened again today. About 30 hours (hmmm.....) after you cleaned it up last. It was actually a bug in the action implementation. Fix in actions/stale#8

@github-actions
Copy link

Labeling this pull request as stale. There has been no activity for 30 days. Remove stale label or comment or this PR will be closed in 7 days.

@github-actions github-actions bot added the no-activity Stale issue or PR that will be automatically closed soon. label Aug 19, 2019
@jrbriggs jrbriggs removed the no-activity Stale issue or PR that will be automatically closed soon. label Aug 19, 2019
@derrickstolee derrickstolee added this to the Demo milestone Aug 20, 2019
@derrickstolee derrickstolee changed the title [WIP] Initial sparse implementation [PR BUILD] Initial sparse implementation [PR BUILD] Aug 20, 2019
@derrickstolee derrickstolee changed the title Initial sparse implementation [PR BUILD] [WIP] Initial sparse implementation [PR BUILD] Aug 20, 2019
@derrickstolee derrickstolee changed the title [WIP] Initial sparse implementation [PR BUILD] Initial sparse implementation [PR BUILD] Aug 20, 2019
@derrickstolee derrickstolee marked this pull request as ready for review August 20, 2019 18:35
@derrickstolee derrickstolee requested review from wilbaker and jrbriggs and removed request for wilbaker August 20, 2019 18:36
@derrickstolee
Copy link
Contributor Author

After rebasing onto a recent master and cloning a big repo, I noticed that I need to prevent the "prefetch everything" code (which wasn't in the previous base). Will fix.

@derrickstolee derrickstolee mentioned this pull request Aug 21, 2019
Merged
derrickstolee added a commit that referenced this pull request Aug 21, 2019
@derrickstolee
Merge pull request #80: CloneVerb: remove --no-mount option
536ae42 
Long term, we will not have a mount, so this feature will
eventually be meaningless. In the meantime, it is only causing
possible problems with hydrating files.

Resolves #26 permanently. In a normal clone, we already resolved
it in #56 by prefetching all files on clone. However, we still
dynamically prefetch on a sparse clone (#54).
@derrickstolee
Initial sparse implementation
ecb213e 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
fix parameter ordering
f27bf46 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
Remove some tests that check prefetch --hydrate
3b8d549 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
Fix the sparse option for 'scalar clone'
b35db24 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee derrickstolee changed the title Initial sparse implementation [PR BUILD] Initial sparse implementation Aug 22, 2019
@derrickstolee derrickstolee mentioned this pull request Aug 22, 2019
Merged
2 tasks
jrbriggs
jrbriggs approved these changes Aug 22, 2019
View reviewed changes
Copy link
Member

@jrbriggs jrbriggs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with suggestions.

Scalar.Build/Scalar.props Show resolved Hide resolved
Scalar/CommandLine/CloneVerb.cs Outdated Show resolved Hide resolved
wilbaker
wilbaker approved these changes Aug 22, 2019
View reviewed changes
Scalar.FunctionalTests/Tests/EnlistmentPerFixture/PrefetchVerbTests.cs Outdated Show resolved Hide resolved
Scalar/CommandLine/CloneVerb.cs Outdated Show resolved Hide resolved
@derrickstolee
Clone: convert --sparse=false to --full-clone
f418dec 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee
PrefetchVerbTests: update test numbers
8631c3c 
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
@derrickstolee derrickstolee mentioned this pull request Aug 22, 2019
Closed
@derrickstolee derrickstolee merged commit 19d745f into microsoft:master Aug 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jrbriggs jrbriggs jrbriggs approved these changes

@wilbaker wilbaker wilbaker approved these changes

Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
Demo
Development

Successfully merging this pull request may close these issues.
3 participants
@derrickstolee @jrbriggs @wilbaker