Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'git fetch' sporadically hangs #747

Open
ghost opened this issue Oct 10, 2017 · 3 comments
Open

'git fetch' sporadically hangs #747

ghost opened this issue Oct 10, 2017 · 3 comments

Comments

@ghost
Copy link

ghost commented Oct 10, 2017

When running 'git fetch' on repositories with a large number of refs, the git client will sometimes stall. On the server, idle git-upload-pack.exe processes remain. Once this happens, this is always reproducible the same local checkout. When fetching from a different source, the problem disappears.

This bug is due to a deadlock that happens when the StandardOutput and StandardInput buffers of the external git-upload-pack process fill. It can be solved by making the read operation on the process output asynchronous.

@ijd02
Copy link

ijd02 commented Jun 28, 2019

Could you please provide more information like source file and line, config settings, error results, error logs, etc?
I'm looking into correcting source code but I'm new to Bonobo & git source.

I have an issue where a fetch operation stalls and the client reports & then hangs. Using TortoiseGit & git for windows 2.14.1.windows.1 as the client and Bonobo 5.1.1 or 6.5.0 with built-in git 1.9.5 or latest git (2.22.0.windows.1) as the server the error reported by the client is:
git.exe fetch -v --progress "origin"

POST git-upload-pack (gzip 1232 to 692 bytes)
POST git-upload-pack (gzip 1432 to 793 bytes)
POST git-upload-pack (gzip 2932 to 1542 bytes)
POST git-upload-pack (gzip 6132 to 3148 bytes)
POST git-upload-pack (gzip 12532 to 6347 bytes)

After this point client hangs.
On the server I see that two processes are running but with no activity, for built-in git 1.9.5 processes are git.exe and git-upload-pack.exe, when 2.22.0 is used two git.exe instances are running.
If I first fetch just master then fetch everything else no error occurs, an easy work around on my local machine but not practical on my Jenkins build infrastructure.

@Ataye
Copy link

Ataye commented Jul 4, 2019

Man, i've been battling this for the last week or so! I ended up putting some logging into the bonobo source and running my compiled dll to try to log where its happeneing etc. I also thought to make the calls asynchronous, but I just had to get this issue resolved. I just ran into it again tonight. I could still clone, and after cloning again fetch is working again, but for how long?

I ended up cloning the bare repo on the server filesystem using --mirror, doing some pruning and pushed to a new repo also created with --mirror, and updating the refs so all origin become local. Then deleted the origin remote and copying the new repo back where the old repo was. This fixed it for most of our devs, except me tonight!

If you are interested in updating to asynchronous, its in the file Bonobo-Git-Server\Bonobo.Git.Server\Git\GitService\GitServiceExecutor.cs, ExecuteServiceByName() method. I've since added a bunch of code so cant tell you which line, yes i can, line 94 using (var process = Process.Start(info)) is the start of the call out to git. If i have time i could do it and create a PR.

@ijd02
Copy link

ijd02 commented Jul 4, 2019

Issue was Bonobo was sending a bunch of data to git via stdIn then then that completed it would read result on stdOut. If data on stdIn caused lots of data to be generated on stdOut then buffer in git would fill and block the process till someone (Bonobo) read stdOut (or maybe a git bug corrupts something when buffer fills?). This never happens because Bonobo would wait till all of stdIn was sent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants