gvfs-helper: better support for concurrent packfile fetches #229
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
derrickstolee
approved these changes
Dec 20, 2019
Teach gvfs-helper to better support the concurrent fetching of the same packfile by multiple instances. If 2 instances of gvfs-helper did a POST and requested the same set of OIDs, they might receive the exact same packfile (same checksum SHA). Both processes would then race to install their copy of the .pack and .idx files into the ODB/pack directory. This is not a problem on Unix (because of filesystem semantics). On Windows, this can cause an EBUSY/EPERM problem for the loser while the winner is holding a handle to the target files. (The existing packfile code already handled simple the existence and/or replacement case.) The solution presented here is to silently let the loser claim victory IIF the .pack and .idx are already present in the ODB. (We can't check this in advance because we don't know the packfile SHA checksum until after we receive it and run index-pack.) We avoid using a per-packfile lockfile (or a single lockfile for the `vfs-` prefix) to avoid the usual issues with stale lockfiles. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
jeffhostetler
force-pushed
the
gvfs-helper-lock-pack-dir
branch
from
ddf2c0b
to
b22f1ec
Compare
|
squashed in the Linux fix. |
This was referenced Dec 26, 2019
derrickstolee
pushed a commit
that referenced
this pull request
Dec 30, 2019
gvfs-helper: better support for concurrent packfile fetches
derrickstolee
added a commit
that referenced
this pull request
Dec 31, 2019
When we create temp files for downloading packs, we use a name based on the current timestamp. There is no randomness in the name, so we can have collisions in the same second. Retry the temp pack names using a new "-<retry>" suffix to the name before the ".temp". This is a follow-up to #229.
derrickstolee
added a commit
to microsoft/scalar
that referenced
this pull request
Dec 31, 2019
These updates allow the gvfs-helper to be more resilient to concurrent pack downloads. See microsoft/git#229 and microsoft/git#231 for more details.
derrickstolee
pushed a commit
that referenced
this pull request
Jan 14, 2020
gvfs-helper: better support for concurrent packfile fetches
derrickstolee
added a commit
that referenced
this pull request
Jan 14, 2020
When we create temp files for downloading packs, we use a name based on the current timestamp. There is no randomness in the name, so we can have collisions in the same second. Retry the temp pack names using a new "-<retry>" suffix to the name before the ".temp". This is a follow-up to #229.
derrickstolee
pushed a commit
that referenced
this pull request
Feb 21, 2020
gvfs-helper: better support for concurrent packfile fetches
derrickstolee
added a commit
that referenced
this pull request
Feb 21, 2020
When we create temp files for downloading packs, we use a name based on the current timestamp. There is no randomness in the name, so we can have collisions in the same second. Retry the temp pack names using a new "-<retry>" suffix to the name before the ".temp". This is a follow-up to #229.
derrickstolee
pushed a commit
that referenced
this pull request
Mar 17, 2020
gvfs-helper: better support for concurrent packfile fetches
derrickstolee
added a commit
that referenced
this pull request
Mar 17, 2020
When we create temp files for downloading packs, we use a name based on the current timestamp. There is no randomness in the name, so we can have collisions in the same second. Retry the temp pack names using a new "-<retry>" suffix to the name before the ".temp". This is a follow-up to #229.
vdye
pushed a commit
that referenced
this pull request
Jul 19, 2023
dscho
pushed a commit
that referenced
this pull request
Aug 8, 2023
dscho
pushed a commit
that referenced
this pull request
Aug 8, 2023
dscho
pushed a commit
that referenced
this pull request
Aug 11, 2023
jeffhostetler
pushed a commit
that referenced
this pull request
Aug 23, 2023
dscho
pushed a commit
that referenced
this pull request
Nov 3, 2023
dscho
pushed a commit
that referenced
this pull request
Nov 3, 2023
dscho
pushed a commit
that referenced
this pull request
Nov 3, 2023
dscho
pushed a commit
that referenced
this pull request
Nov 8, 2023
dscho
pushed a commit
that referenced
this pull request
Nov 14, 2023
dscho
pushed a commit
that referenced
this pull request
Nov 20, 2023
vdye
pushed a commit
that referenced
this pull request
Feb 27, 2024
dscho
pushed a commit
that referenced
this pull request
Apr 23, 2024
dscho
pushed a commit
that referenced
this pull request
Apr 23, 2024
dscho
pushed a commit
that referenced
this pull request
Apr 23, 2024
dscho
pushed a commit
that referenced
this pull request
Apr 24, 2024
dscho
pushed a commit
that referenced
this pull request
Apr 29, 2024
dscho
pushed a commit
that referenced
this pull request
May 14, 2024
dscho
pushed a commit
that referenced
this pull request
May 14, 2024
dscho
pushed a commit
that referenced
this pull request
Jun 3, 2024
dscho
pushed a commit
that referenced
this pull request
Jul 17, 2024
dscho
pushed a commit
that referenced
this pull request
Jul 17, 2024
dscho
pushed a commit
that referenced
this pull request
Jul 17, 2024
dscho
pushed a commit
that referenced
this pull request
Jul 18, 2024
mjcheetham
pushed a commit
that referenced
this pull request
Jul 23, 2024
dscho
pushed a commit
that referenced
this pull request
Jul 25, 2024
mjcheetham
pushed a commit
that referenced
this pull request
Jul 29, 2024
dscho
pushed a commit
that referenced
this pull request
Sep 18, 2024
dscho
pushed a commit
that referenced
this pull request
Sep 24, 2024
dscho
pushed a commit
that referenced
this pull request
Oct 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Teach gvfs-helper to better support the concurrent fetching of the
same packfile by multiple instances.
If 2 instances of gvfs-helper did a POST and requested the same set of
OIDs, they might receive the exact same packfile (same checksum SHA).
Both processes would then race to install their copy of the .pack and
.idx files into the ODB/pack directory.
This is not a problem on Unix (because of filesystem semantics).
On Windows, this can cause an EBUSY/EPERM problem for the loser while
the winner is holding a handle to the target files. (The existing
packfile code already handled simple the existence and/or replacement
case.)
The solution presented here is to silently let the loser claim
victory IIF the .pack and .idx are already present in the ODB.
(We can't check this in advance because we don't know the packfile
SHA checksum until after we receive it and run index-pack.)
We avoid using a per-packfile lockfile (or a single lockfile for
the
vfs-prefix) to avoid the usual issues with stale lockfiles.Signed-off-by: Jeff Hostetler jeffhost@microsoft.com