[GoogleCode Issue 457] Push over git protocol hangs in msysGit

[childnode]

This is a resurrected issue from the old issue tracker ... still existing .. still annoying.

patch
0c499ea60f
marked to be cause of this push hangup as mentioned by johannes in comment#4

marked as "WontFix" at Apr 25, 2010 due to "nobody wants to fix" ;/
possible patch mentioned in comment#20 / comment#41

most details of problem analysys can be found in comment#35

File that needs patch nowadays (I think): https://github.com/msysgit/git/blob/master/send-pack.c

[dscho]

@childnode please feel free to take ownership of this issue and work towards its conclusion. In the links you referenced, there is a ton of information about the problem and the suggested ways to debug/fix.

[childnode]

I would, If I could do it.. unfortunatly my C skills are quite bad .. hacking and disabling the sideband would not solve the core problem and I expect you guys won't accept this patch, else you have had done this yourself in a minute, right?! ;)

[t-b]

Does anyone have a self-contained script (sh or batch) to reproduce the bug? This would be the obvious first step towards fixing the bug.

[dscho]

@childnode I understand. Unfortunately, the issue will need to wait, then, until somebody comes along driving the fix.

[dscho]

The issue can wait while being closed to protect my time (because it would otherwise see me, not @childnode, look at it every once in a while)

[slothbag]

I don't understand how this bug can be ignored for so long. Gits basic push functionality does not work even on the simplest of configurations.

I assume most people are hex editing the binary like me to make it work.

[t-b]

First step is still
#101 (comment) as not everyone encounters that bug.

Btw. hex editing binaries from FLOSS software sounds really awful.

[dscho]

@slothbag I guess it is not fixed at all because people like you make it work that way.

[cphillips83]

Using VMWare ESXi, running 2 machines with Win2k8. Install git on both machines from git-scm.com.

Machine A (open up port 9418, IP:10.0.0.220)
run cmd
mkdir c:\git\bareserver
cd c:\git\bareserver
git --bare init
git daemon --reuseaddr --base-path=C:\git --export-all --verbose --enable=receive-pack

Machine B (IP 10.0.0.225):
run cmd
mkdir C:\git
cd C:\git
git clone git://10.0.0.220/bareserver
cd bareserver
echo test >> readme.txt & git add . & git commit -m "test" & git push

Result:
[master 9447645] test
1 file changed, 1 insertion(+)
Counting objects: 20, done.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (18/18)

Stuck sitting at writing objects, never completes (I have a few more objects from running this test a few times).

[cphillips83]

Leaving the above comment in tack as that was the original way I discovered the problem.

I was able to reproduce this on another box with a straight Win7 install using 1.8.5.2.msysgit.0 and the following batch file.

mkdir c:\git\hangserver
cd c:\git\hangserver
git --bare init
start cmd /c git daemon --reuseaddr --base-path=C:\git --export-all --verbose --enable=receive-pack

git clone git://localhost/hangserver hangclient
cd hangclient
echo test >> readme.txt & git add . & git commit -m "test" & git push

[cphillips83]

@t-b I have added a script above this comment that can reproduce it on a Win7 x64 machine (home premium) i5 2300 2.8ghz quad

[cphillips83]

I also see this in the task manager when running my script. Is this normal to have it launch 6 git.exes and 2 daemons?

I have killed all copies of this and ran the script again with the same result.

[cphillips83]

Not to keep spamming this thread, but I just came across something strange, while trying to attach the VS2012 debugger to all 6 git.exe at once, it finished properly (even tho it was already sitting there for 5+ minutes while I tried to investigate).

I can not consistently reproduce this. While attaching the debugger does have some sort of impact, VS2012 never successfully attached and when you end task devenv, the git push either fails or succeeds. I'm willing to put the time in to debug this as its a critical show stopper for me right now, but I have no idea where to start.

I'll update this comment periodically until someone comes along.

[t-b]

@cphillips83 Thanks! At last I can reproduce the problem.
Using your snippets from #101 (comment) I can reproduce the hanging on windows 7 x64.

[dscho]

Just a hunch: could this be related to #70?

[t-b]

@dscho not sure that the OP in #70 wanted to do!?

Current state of affairs can be explained with applying the following patch:

diff --git a/wrapper.c b/wrapper.c
index 6a015de..ab04ac5 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -155,7 +155,9 @@ ssize_t xwrite(int fd, const void *buf, size_t len)
 {
        ssize_t nr;
        while (1) {
+               warning("xwrite0 fd=%d, buf=%s, len=%d", fd, (char*) buf, len);
                nr = write(fd, buf, len);
+               warning("xwrite1 nr=%d", nr);
                if ((nr < 0) && (errno == EAGAIN || errno == EINTR))
                        continue;
                return nr;

Recompile, start a daemon
git-daemon.exe --reuseaddr --base-path=/C/git --export-all --verbose --enable=receive-pack
make some commits and push them with
git push
results in the output

$ /h/mingwGitDevEnv-installed-v2/packages/mingw32-git/git-1.8.4-2/build/git.exe push
warning: xwrite0 fd=4, buf=0030git-receive-pack /hangserver, len=48
warning: xwrite1 nr=48
warning: xwrite0 fd=4, buf=009ca26b87c65946d4dbb146082f5688290892eb4eb5 34cff47c67923a47ad610a82a3d58c6604623c
e8 refs/heads/master, len=156
warning: xwrite1 nr=156
warning: xwrite0 fd=4, buf=0000, len=4
warning: xwrite1 nr=4
warning: xwrite0 fd=8, buf=^a26b87c65946d4dbb146082f5688290892eb4eb5
°[T☻, len=42
warning: xwrite1 nr=42
warning: xwrite0 fd=8, buf=34cff47c67923a47ad610a82a3d58c6604623ce8

°[T☻, len=41
warning: xwrite1 nr=41
Counting objects: 5, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (2/2), done.
warning: xwrite0 fd=1, buf=PACK, len=253

with git push not returning.

Which means that write does not return as already found in https://code.google.com/p/msysgit/issues/detail?id=457#c35.

[dscho]

@t-b It was just a shot in the dark, as #70 purported that the socket was closed prematurely (which in my mind could be a valid explanation why the write() fails). Dunno.

I guess that this one will be very tricky to debug, and not without instrumenting the daemon, and the close() function, as well and adding timestamps to the debug log.

Sorry that I cannot be more helpful!

[pango]

While migrating our git Server from Solaris to Red Hat Linux we ran into exactly this issue. Our Win 7 Clients could no longer push. Client and server side of git hang after the Message "Writing objects: 100% (12/12)". We wasted 3 days in analysis and trying to find a workaround. We tried differend workarounds and versions of git but nothing seemed to work.
Interessting: We switched from GIT Deamon to SSH and were able to push with errors.
The push seemed to work but it ended with:
send-pack: protocol error: bad band #117
fatal: The remote end hung up unexpectedly

Finaly we found a hack that made git push working properly again.
http://billauer.co.il/blog/2012/10/git-pull-windows-freeze-receive-pack/
This is not the best solution, but it seems to works.

I would strongly reccomend you guys to solve this bug. The Issue 457 in msysgit is known for 4 years now.

[t-b]

1.) Did you get any further information from your analysis?

2.) The issue here deals with git daemon only. Therefore your problem with the ssh protocol belongs either on the mailing list or in a separate issue. In case you file a separate issue please provide enough information so that we can reproduce it. AFAIK no problems are known with the ssh protocol.

[kusma]

@pango Unfortunately, knowing about the bugs existence for 4 years haven't helped much in figuring out why it happens. And people affected seems to have had mixed results with disabling side-band-64k, so we're not too confident about this as a work-around.

[dscho]

I would hardly reccomend you guys to solve this bug.

First of all, I believe that you meant "strongly", not "hardly" (which is the opposite).

Second: you do understand that you guys have no support contract with us? We already worked very hard, without getting any money from you or anybody else, to get things where they are right now. Asking for even more, for free, without even the offer to do cooperate on solving this bug, well, you know, this is kind of counter-productive?

The Issue 457 in msysgit is known since 4 years now.

Again, I assume you mean "for" instead of "since". And yes, the issue is open for four years. But apparently it is not an issue for any of the people who actually contribute fixes to Git for Windows.

I would strongly suggest to try motivating in a more positive way any person with enough expertise who could help you. Taking myself as example, after reading so much negativity and feeling personally insulted (despite putting in so many thousands of hours I get criticized!), I would only consider to work on the bug -- which is not a bug I have to deal with at all! -- after an apology and a contract that makes sure that I get paid to solve your problem.

I seriously hope that this clarifies the situation.

[slothbag]

@dscho This is a common theme surrounding open source projects, people complaining about bugs in something they got for free and the contributors feeling used and unappreciated. I think its something you either have to block out or end up giving up because of.

I certainly didn't expect anyone to fix this issue based on my comment, I was simply stating that i'm surprised it (the bug) has been around this long. I wish I had the expertise (and time) to fix it, but I don't.

Would any of the "sufferers" of this bug like to join me in creating a bounty to get this fixed? I can start by pledging $100 (in bitcoin) for a proper fix.

[cphillips83]

@dscho This is where I have to disagree with you. The fact that it is open
source doesn't mean you get to fire this excuse every time something
doesn't work. This is an amazing project and there isn't a single
alternative project for git on windows that I would even consider at this
point. That being said, the project has raised its self to 'stardom" so to
speak and that means its bar is raised far above what the casual open
source project is.

You seem to get defensive every time someone comments that this bug has
been open for so long and everyone is right, its mind boggling that no
contributors have run in to this issue. One of the few ways you can host
git on windows simply does not work. Its not like anyone is sitting here
saying that if certain stars align and the earth is slightly off its axis
this bug happens. I gave an absolutely simple cut and dry bat file that
will fail every time and it renders hosting git on windows completely
useless (my work around was to host it on a centos box instead).

I have a lot more to say on this matter, but this isn't the place for this
sort of discussion so it will be my one and only comment on it. After
losing days of time on this issue, reading a 4 year history on this bug, to
find it closed when it was never fixed, then produce a perfectly workable
example of it failing everytime, to then read you need to motivate us, I
feel like I simply wasted my time.

On Thu, Apr 10, 2014 at 10:01 AM, slothbag notifications@github.com wrote:

@dscho https://github.com/dscho This is a common theme surrounding open
source projects, people complaining about bugs in something they got for
free and the contributors feeling used and abused. I think its something
you either have to block out or end up giving up because of.

I certainly didn't expect anyone to fix this issue based on my comment, I
was simply stating that i'm surprised it (the bug) has been around this
long. I wish I had the expertise (and time) to fix it, but I don't.

Would any of the "sufferers" of this bug like to join me in creating a
bounty to get this fixed? I can start by pledging $100 (in bitcoin) for a
proper fix.

Reply to this email directly or view it on GitHubhttps://github.com//issues/101#issuecomment-40084898
.

[imanuelcostigan]

@dscho are you looking for an "incentive" to get this fixed? If so, what is it? Everyone here (me included) doesn't have the expertise or knowledge of git's internals to patch while some are resorting to hacks.

[t-b]

One missing piece of information is this post which explains very good the reasons for this bug.

Having spent more than a full day trying to understand the bug, I've come, together with all the bits and pieces found elsewhere by my predecessors, to the following conclusion:

• The stuck push operation is due to a deadlock in the msvcrt.dll. As we can not influence that directly, we must prevent it from happening in our code. All proper solutions I can think of now require a huge amount of work as we would have to start distinguishing between a file descriptor steeming from a socket and all others. Not only would that be a lot of work, it would also multiply our diff with upstream git. Something which should never be our target.
• Turning the sideband capability just off, is also not a proper solution. Many people, including me, don't use the dump git protocol at all, therefore these people would suffer a feature degradation from turning it statically off. It would also make the default behaviour on windows different than on *nix.

Having said that all, I propose to merge my sideband-bug branch.

Some comments:

• The patch does not introduce changes to the default behaviour.
• Makes the compile-it-yourself/hack-the-binary work arounds unnecessary.
• I'll ask upstream for input on this first windows-only configuration variable.
• The git documentation is not built from source, therefore the changes will not show up in the shipped documentation. This problem will be in the future solved by the work currently done in mingwGitDevEnv.

[cphillips83]

This is pretty excellent and what a lot of us were looking for. We all know
its not proper but its been suggested before to give us an option to
disable side band so we don't have to edit the binary. This to me would be
a perfect acceptably solution until someone comes along to figure it out.

Unfortunately this bug was such a nightmare for me that I ended up writing
my own extremely light weight git clone project (probably shouldn't even
call it that) to solve my issues. I didn't need all the powerful features
of Git, but I needed to get a specific revision of a commit and I also
needed it to handle binaries more efficiently (I use this for product
release management).

Thanks for your efforts Thomas

+1

On Thu, May 8, 2014 at 4:29 PM, Thomas Braun notifications@git.luolix.topwrote:

One missing piece of information is thishttps://groups.google.com/forum/#!msg/msysgit/at8D7J-h7mw/PM9w-d41cDYJpost which explains very good the reasons for this bug.

Having spent more than a full day trying to understand the bug, I've come,
together with all the bits and pieces found elsewhere by my predecessors,
to the following conclusion:

• The stuck push operation is due to a deadlock in the msvcrt.dll. As
  we can not influence that directly, we must prevent it from happening in
  our code. All proper solutions I can think of now require a _huge_amount of work as we would have to start distinguishing between a file
  descriptor steeming from a socket and all others. Not only would that be a
  lot of work, it would also multiply our diff with upstream git. Something
  which should never be our target.
• Turning the sideband capability just off, is also not a proper
  solution. Many people, including me, don't use the dump git protocol at
  all, therefore these people would suffer a feature degradation from turning
  it statically off. It would also make the default behaviour on windows
  different than on *nix.

Having said that all, I propose to merge my sideband-bughttps://github.com/t-b/git/tree/sideband-bugbranch.

Some comments:

• The patch does not introduce changes to the default behaviour.
• Makes the compile-it-yourself/hack-the-binary work arounds
  unnecessary.
• I'll ask upstream for input on this first windows-only configuration
  variable.
• The git documentation is not built from source, therefore the
  changes will not show up in the shipped documentation. This problem will be
  in the future solved by the work currently done in mingwGitDevEnv.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/101#issuecomment-42602186
.

[toshiyuki-ogawa]

@t-b I agree with you. Thanks for your great suggestion.

we would have to start distinguishing between a file descriptor stemming from a socket and all others

I think so too.
In #70, I had to write code for telling whether a file descriptor is stemmed from socket or not. In my experience, I think that a windows socket attribute was not handled properly on spawned process.