Pull performance

[XJPan-Jeff]

I met a problem with pull performance, can you please shed some lights on it ?

1. Our setup
   We are migrating a P4 repo to Azure repo. It is a huge huge code base, with 10GB source code in C++ and 80GB of binary model files. All were stored and version managed in P4, now we are moving them to Azure repo. We are leveraging -

   • Scalar clone to reduce the first clone time
   • LFS for big binary files
   • Sparse checkout

2. The problem
   The initial setup works. The issue is - when I git pull after a few commits , there are 15K+ objects (around 1GB data) to send from remote. But it is estimated to have <1MB data in the new commits only.

Don't understand why so many objects are send via pull... Any idea?

[XJPan-Jeff]

Here is the screenshot fyi

[derrickstolee]

If you are intending to use Scalar to reduce clone sizes, then you need to install the microsoft/git fork of Git in order to get the ability to use the "GVFS Protocol" against Azure Repos. The GVFS Protocol is Azure Repos' equivalent of Git's partial clone, which you can see showing up in your error output:

warning: filtering not recognized by server, ignoring

I recommend installing the latest full release of microsoft/git and trying your scalar clone again.

[XJPan-Jeff]

Thank you Stolee. The issue seems resolved as per test in these 2 days after installing microsoft git. Many thanks!

[XJPan-Jeff]

Still problematic... The below screenshot shows 3 git pull commands -

• The 1st one has changes to fetch, so it takes time. The 2nd is executed right after the 1st pull, it is quite fast as expected. Then after 2 hours, I ran the 3rd pull, it also take very long time.
• And, why it shows "Prefetch 0 (1970-01-01 00:00:00 +0000)", the year zero of IT ages?
  Any insight?

[derrickstolee]

• And, why it shows "Prefetch 0 (1970-01-01 00:00:00 +0000)", the year zero of IT ages?

That is going to be the critical issue. The prefetching code is looking at your pack-file names to detect a timestamp to send to the server. The client is failing to recognize that timestamp, so is asking for all of the data again.

Please run scalar diagnose, which will produce a .zip file with a lot of data. Instead of sending the full logs, just unzip it and attach two files to a reply:

• packs-local.txt
• packs-cached.txt

That will help to determine what is going on here.

[XJPan-Jeff]

packs-local.txt
packs-cached.txt

Thanks Stolee. Please find the 2 files you asked. Please note that it is not always reproducible, most of the time, it prefetches from the last update timestamp.

I am using git version 2.33.0.vfs.0.0, no cache server, feature: fsmonitor--daemon, on Windows.

[derrickstolee]

Here is the critical part of the packs-cached.txt that I was looking for:

prefetch-1634893078-373fcd3e095b3a59dbb4b530119811eb91c72943.idx               16506400
prefetch-1634893078-373fcd3e095b3a59dbb4b530119811eb91c72943.pack              88742801
prefetch-1635131207-0283309e6753d84a144878f2afb2bb1f76e0fd1d.idx                  28680
prefetch-1635131207-0283309e6753d84a144878f2afb2bb1f76e0fd1d.keep                     0
prefetch-1635131207-0283309e6753d84a144878f2afb2bb1f76e0fd1d.pack                789833
vfs-10b1a81bfc04a06ce688c9a6eb8672fa4c6233e1.idx                                 113072
vfs-10b1a81bfc04a06ce688c9a6eb8672fa4c6233e1.pack                             200424419

1. You don't have many prefetch- packs, which means that's the data you are re-downloading, but only if the most-recent prefetch- pack-file is missing.
2. You don't have a multi-pack-index file in that directory, which is strange. You must not have run maintenance recently.
3. There is no evidence in packs-cached.txt or packs-local.txt that something else repacked the prefetch- packs away into something else.

This is leading me to thing that something is deleting your data. Is it possible that this D:/.scalarCache/id_d08ee092-88d7-4047-aa58-b9bce60d1562/pack directory is being deleted somehow?

[XJPan-Jeff]

So sharp! Yes I deleted the folder you mentioned as to clean up my D drive. I did not realize it is something still required by Scalar. Will it be better to put the cache to .git folder? BTW, What do you mean "You don't have a multi-pack-index file in that directory, which is strange. You must not have run maintenance recently"? From: Derrick Stolee ***@***.***> Sent: Monday, October 25, 2021 8:50 PM To: microsoft/scalar ***@***.***> Cc: PAN Xiaojin ***@***.***>; Author ***@***.***> Subject: Re: [microsoft/scalar] Pull performance (Discussion #516) You don't often get email from ***@***.******@***.***>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> This email is not from Hexagon's Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email. Here is the critical part of the packs-cached.txt that I was looking for: prefetch-1634893078-373fcd3e095b3a59dbb4b530119811eb91c72943.idx 16506400 prefetch-1634893078-373fcd3e095b3a59dbb4b530119811eb91c72943.pack 88742801 prefetch-1635131207-0283309e6753d84a144878f2afb2bb1f76e0fd1d.idx 28680 prefetch-1635131207-0283309e6753d84a144878f2afb2bb1f76e0fd1d.keep 0 prefetch-1635131207-0283309e6753d84a144878f2afb2bb1f76e0fd1d.pack 789833 vfs-10b1a81bfc04a06ce688c9a6eb8672fa4c6233e1.idx 113072 vfs-10b1a81bfc04a06ce688c9a6eb8672fa4c6233e1.pack 200424419 1. You don't have many prefetch- packs, which means that's the data you are re-downloading, but only if the most-recent prefetch- pack-file is missing. 2. You don't have a multi-pack-index file in that directory, which is strange. You must not have run maintenance recently. 3. There is no evidence in packs-cached.txt or packs-local.txt that something else repacked the prefetch- packs away into something else. This is leading me to thing that something is deleting your data. Is it possible that this D:/.scalarCache/id_d08ee092-88d7-4047-aa58-b9bce60d1562/pack directory is being deleted somehow? - You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fmicrosoft%2Fscalar%2Fdiscussions%2F516%23discussioncomment-1532185&data=04%7C01%7C%7C650b23612321420405a208d997b5f770%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637707630067235903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZlImkSuCYXp6qgrrvz0e7VA77OU7L5mQ3Vnc4sYuM4A%3D&reserved=0>, or unsubscribe<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fnotifications%2Funsubscribe-auth%2FARO7KSJBU26DXYTGHE75PCLUIVG7XANCNFSM5GBGWHBQ&data=04%7C01%7C%7C650b23612321420405a208d997b5f770%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637707630067245856%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FEs2hUQUwa1SLEMdBAcEFjWZa652e8svx0%2Fh9jt6mKs%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C650b23612321420405a208d997b5f770%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637707630067245856%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=knoQYHOnhcoaEvvT4ZuC5fLwWdfXZb1o7lp6jM8snzI%3D&reserved=0> or Android<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C650b23612321420405a208d997b5f770%7C1b16ab3eb8f64fe39f3e2db7fe549f6a%7C0%7C0%7C637707630067255815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=f1swCq10yHwR6Z%2F%2B1wOBnC0BUwZ05e4k9CTD3ouNN8k%3D&reserved=0>.

[derrickstolee]

So sharp! Yes I deleted the folder you mentioned as to clean up my D drive. I did not realize it is something still required by Scalar. Will it be better to put the cache to .git folder?

We use the .scalarCache directory to store object data so you can have multiple scalar clone directories but de-duplicate the object database into one location.

If you want this data in a different location, then you can use the --local-cache-path option during your clone.

BTW, What do you mean "You don't have a multi-pack-index file in that directory, which is strange. You must not have run maintenance recently"?

scalar clone initializes background maintenance, including the incremental-repack task, which generates a multi-pack-index file. This speeds up object lookups to be logarithmic in your number of objects without being linear in the number of pack-files. This task should run nightly.

[XJPan-Jeff]

Many thanks!!

With regards to the maintenance, can you please additionally help me clarify more?

• how can I check the backend maintenance and enable it if it is not running as expected?
• In which scenario, the maintenance will stop?
• Will Scalar maintenance job fetch the LFS objects?

Sorry too many questions, it is super helpful for me to understand Scalar + GIT. Super appreciate your explaination!!

[derrickstolee]

You can see if your repo is in the list from git config --global --list maintenance.repo. You can also always be sure it is enabled by running git maintenance start in your repo. Maintenance won't stop unless you run git maintenance stop (to shut down all maintenance everywhere) or git maintenance unregister (to remove the current repo from the maintenance list).

The maintenance jobs will not fetch LFS objects. LFS objects are faulted in during a git checkout, so they specifically are not fetched in advance. Part of the point is to avoid downloading every version of those large blobs from history.