Cygwin error during git push azure master

[tjanczuk]

/cc @davidebbo

1. Create a new site from within an existing local repo azure site create --git. The local repo size is ~3.5MB
2. Set the site mode to Standard (via portal) - not clear if this is relevant.
3. Try to deploy with git push azure master.

Expected:

Kudu repo updated, files copied to wwwroot, site can be accessed in browser.

Actual:

C:\projects\libreweb>git push azure master
Password for 'https://mobilechapters@mobilechapters.scm.azurewebsites.net':
Counting objects: 2175, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1925/1925), done.
Writing objects: 100% (2175/2175), 3.42 MiB | 501.00 KiB/s, done.
Total 2175 (delta 1407), reused 284 (delta 160)
remote:       0 [main] us 0 init_cheap: VirtualAlloc pointer is null, Win32 error 487
remote: AllocationBase 0x9D0000, BaseAddress 0xA00000, RegionSize 0x150000, State 0x2000
remote: D:\Program Files (x86)\Git\bin\sh.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 0
To https://mobilechapters@mobilechapters.scm.azurewebsites.net/mobilechapters.git

Subsequent git push azure master claims the Kudu repo is up to date:

C:\projects\libreweb>git push azure master
Password for 'https://mobilechapters@mobilechapters.scm.azurewebsites.net':
Everything up-to-date

Yet navigating to the site in browser yields the autogenerated landing page - it is clear files have not been copied to wwwroot.
@Gissues:{"order":77.88461538461526,"status":"backlog"}

[tjanczuk]

Making a dummy change in local git repo and re-running git push azure master appears to complete the process of copying files from the Kudu repo to wwwroot.

[davidebbo]

This is an issue that we had a while back, and seems to be a general msysgit thing that people run into occasionally (not related to WAWS). e.g. see various references to it. From our experience, this happens at most once on a given server, so I suspect you won't hit this again (though you may get moved to a different server later).

One advice given in a few places is to rebase msys-1.0.dll, though the exact rebase address varies depending on who you ask! e.g. see here vs there.

At the time (around April I think), we ended up resolving the issue by rebasing using this command when the worker starts:

"%ProgramFiles(x86)%\Git\bin\rebase.exe" -b 0x30000000 "%ProgramFiles(x86)%\Git\bin\msys-1.0.dll"

And the issue completely disappeared at that point.

Right now, I don't understand what is causing it to happen again. What makes it tricky is that it's very hard to repro, and no one truly seems to understand the root of the issue.

[tjanczuk]

It ranks 387 on my list of problems to deal with (the preceding 386 are mostly about IE Mobile), so no big deal. I just thought you would want to know.

[davidebbo]

Yes, we do want to know since we thought we had it nailed. :(

[davidebbo]

There have been no other reports of this. Closing it as a freak incident :)

[AlanParr]

I am now experiencing this issue on a website that was created just today, my first push was successful although the deployment failed for unrelated reasons (missing assemblies). I have attempted to push several more times after making arbitrary changes to the project and just get Everything up-to-date every time.
Do you need me to open a new issue or can we re-use this one?

[suwatch]

@AlanParr if you retry, is it successful?

[AlanParr]

Have retried several times with no success.

[davidebbo]

Do you mean that you get the cygwin error every time, or just the message saying it's up to date? If the latter, just try to push an empty commit, which should unblock it.

[AlanParr]

Just the everything up to date error. Sorry, didn't see that there was another error associated with this issue.

[davidebbo]

Yes, the primary issue here is the cygwin error, with the 'up to date' part being a secondary side effect (see initial post). So if you never saw the cygwin error, it is definitely not related and would be better discussed separately.

[AlanParr]

I'll open a new issue.

[Petah]

Getting this multiple times...

c:\work>git push azure develop
Password for 'https://......@..............scm.azurewebsites.net:443':
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 262 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Updating branch 'develop'.
remote: Updating submodules.
remote: Preparing deployment for commit id 'c0e7a5942c'.
remote: Running custom deployment command...
remote: Running deployment command...
remote: Handling Basic Web Site deployment.
remote: Node.js versions available on the platform are: 0.6.17, 0.6.20, 0.8.2, 0.8.19, 0.8.26, 0.8.27, 0.8.28, 0.10.5, 0.10.18, 0.10.21, 0.10.24, 0.10.26, 0.10.28, 0.10.29, 0.10.31, 0.10.32, 0.12.0, 0.12.2.
remote: Selected node.js version 0.12.0. Use package.json file to choose a different version.
remote: Updating iisnode.yml at D:\home\site\wwwroot\iisnode.yml
remote: m.AllocationBase 0x0, m.BaseAddress 0x68570000, m.RegionSize 0x260000, m.State 0x10000
remote: An error has occurred during web site deployment.
remote: D:\Program Files (x86)\Git\bin\cat.exe: *** Couldn't reserve space for cygwin's heap (0x68570000 <0x980000>) in child, Win32 error 0
remote: getting node version failed
remote:
remote: Error - Changes committed to remote repository but deployment to website failed.
To https://....@.................scm.azurewebsites.net:443/...............git
   412caa2..c0e7a59  develop -> develop

And some variations:

remote: D:\Program Files (x86)\Git\bin\bash.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 0
remote: m.AllocationBase 0x0, m.BaseAddress 0x68570000, m.RegionSize 0x260000, m.State 0x10000
remote: D:\Program Files (x86)\Git\bin\dirname.exe: *** Couldn't reserve space for cygwin's heap (0x68570000 <0x990000>) in child, Win32 error 0
remote: m.AllocationBase 0x0, m.BaseAddress 0x68570000, m.RegionSize 0x260000, m.State 0x10000
remote: D:\Program Files (x86)\Git\bin\uname.exe: *** Couldn't reserve space for cygwin's heap (0x68570000 <0xB00000>) in child, Win32 error 0

[brainded]

+1

Tried to do a push and just got the cygwin error as well. Any subsequent pushes tell me that "Everything is up-to-date"

[petetnt]

I switched my (working) webapp to another just now and on the initial push I ran to the same error.

remote:       0 [main] us 0 init_cheap: VirtualAlloc pointer is null, Win32 error 487
remote: AllocationBase 0x0, BaseAddress 0x68570000, RegionSize 0x4A0000, State 0x10000
remote: D:\Program Files (x86)\Git\bin\sh.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 0
To https://xxx.scm.azurewebsites.net:443/xxx.git
 * [new branch]      master -> master

Like @brainded, I also get "Everything up-to-date" for all pushes, even with --force

Doing some dummy changes eventually fixed the thing though.

[brainded]

What I ended up doing to resolve the issue is spinning up a new slot to swap with and deploying the whole app again. This was the one time I've ever encountered this with the last year of using azure to deploy almost every single day. So rare indeed.

[davidebbo]

We're hoping to move to the newest msysgit 2.5 in the coming months, and hopefully it will address this issue.

[Praburaj]

I'm running into this in our azure websites too.

[davidebbo]

Keeping issue open.

[davidebbo]

I added this to the wiki: https://github.com/projectkudu/kudu/wiki/Known-issues#you-may-get-occasional-couldnt-reserve-space-for-cygwins-heap-errors-when-git-pushing, so all the relevant information is in one place.

[jeroenseegers]

Just ran into this issue while deploying as well. Any updates on the status of this?

Edit: For good measure, below are the two errors I encountered;

First push:

Password for 'https://...@.............scm.azurewebsites.net:443':
Counting objects: 42, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (42/42), done.
Writing objects: 100% (42/42), 9.62 KiB | 0 bytes/s, done.
Total 42 (delta 30), reused 0 (delta 0)
remote:       0 [main] us 0 init_cheap: VirtualAlloc pointer is null, Win32 error 487
remote: AllocationBase 0xB40000, BaseAddress 0xBC0000, RegionSize 0x40000, State 0x2000
remote: D:\Program Files (x86)\Git\bin\sh.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 0
To https://...@.............scm.azurewebsites.net:443/....git
   8e68cb4..235fed3  master -> master

Second push after another commit:

Password for 'https://...@.............scm.azurewebsites.net:443':
Counting objects: 7, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 580 bytes | 0 bytes/s, done.
Total 7 (delta 5), reused 0 (delta 0)
remote: Updating branch 'master'.
remote: Updating submodules.
remote: Preparing deployment for commit id '792caf04d8'.
remote: Running custom deployment command...
remote: Running deployment command...
remote:       0 [main] us 0 init_cheap: VirtualAlloc pointer is null, Win32 error 487
remote: AllocationBase 0xA10000, BaseAddress 0xA20000, RegionSize 0x171000, State 0x1000
remote: D:\Program Files (x86)\Git\bin\bash.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 0
remote:
remote: Error - Changes committed to remote repository but deployment to website failed.
To https://...@.............scm.azurewebsites.net:443/....git
   235fed3..792caf0  master -> master

Using the "workaround" from the documentation seems to have worked.

[EddieFarah]

It seems that restarting the website (through azure dashboard) and then pushing/deploying again worked for me.

[M-rc-l]

FYI; deployment on our webapp started to act weird yesterday evening. In the late afternoon the deployments were still working, but eventually we got weird nmp errors (like Error: EPERM: operation not permitted, rename).

We've decided to delete the node_modules folder and execute an npm cache clean (which first failed but then worked).

However, after that the couldn't reserver space for cygwin's heap errors started showing up at various places in the deployment sequence (on executing bash deploy.sh, on updating submodules and even when executing a ls -a through kuduexec).

Neither start/stop nor restart of the web app nor various re-deployments (both via azure cli deployment redeploy and pushing dummy commits), have solved this problem for us.

Any advice on how to solve this will be much appreciated...

Best regards,
Marcel

p.s. If there are no other steps we can try, we are planning to remove and recreate the wep app... no problem since we are still in the testfase. However getting stuck on deployment like this doesn't really gives us much confidence that we can use Azure for production...

[davidebbo]

Normally, this error happens once (or sometimes once per tool) and then goes away on retry.

We're in the process of upgrading the version of git running in Web Apps, which hopefully will resolve this.

[M-rc-l]

Thanks for the reply @davidebbo.

I left the instance alone for a couple of hours and just tried again. No more cygwin errors - yay!

The first try did fail however - with an 'xyz' appears as both a file and a directory, stackTrace: at LibGit2Sharp.Core.Ensure.HandleError(Int32 result) error. So I guess the previous failures must have left the repo in some inconsistent state.

After removing everything in d:/home/site/repository and re-initializing an git repo, the deployment started perfectly fine. However midway of npm install we're back to the npm ERR! EINVAL: invalid argument, rename ... error (will post complete error below, just in case you're interested).

So I'm writing this (web app) instance off and will create and switch to a new one.

------ fyi snippet from log ------

npm ERR! Windows_NT 6.2.9200
npm ERR! argv "d:\\Program Files (x86)\\nodejs\\4.2.4\\node.exe" "D:\\Program Files (x86)\\npm\\2.14.12\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v4.2.4
npm ERR! npm  v2.14.12
npm ERR! path D:\home\site\repository\node_modules\grunt-contrib-cssmin\node_modules\maxmin\node_modules\pretty-bytes\node_modules\meow\node_modules\redent\node_modules\indent-string\node_modules\repeating\node_modules\is-finite\package.json.f47e5fb9605e791b223dbeab8189d496
npm ERR! code EINVAL
npm ERR! errno -4071
npm ERR! syscall rename

npm ERR! EINVAL: invalid argument, rename 'D:\home\site\repository\node_modules\grunt-contrib-cssmin\node_modules\maxmin\node_modules\pretty-bytes\node_modules\meow\node_modules\redent\node_modules\indent-string\node_modules\repeating\node_modules\is-finite\package.json.f47e5fb9605e791b223dbeab8189d496' -> 'D:\home\site\repository\node_modules\grunt-contrib-cssmin\node_modules\maxmin\node_modules\pretty-bytes\node_modules\meow\node_modules\redent\node_modules\indent-string\node_modules\repeating\node_modules\is-finite\package.json'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     D:\home\site\repository\npm-debug.log
deploy.sh: line 106: 12064 Signal 112              (core dumped) "D:\Program Files (x86)\nodejs\4.2.4\node.exe" "D:\Program Files (x86)\npm\2.14.12\node_modules\npm\bin\npm-cli.js" install
c:\Program Files (x86)\SiteExtensions\Kudu\51.50119.2041\bin\scripts\starter.cmd bash deploy.sh, stackTrace:    at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args)
   at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args)
   at Kudu.Core.Deployment.Generator.ExternalCommandBuilder.RunCommand(DeploymentContext context, String command, String message)
   at Kudu.Core.Deployment.Generator.CustomBuilder.Build(DeploymentContext context)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Kudu.Core.Deployment.DeploymentManager.<Build>d__29.MoveNext()

[davidebbo]

@M-rc-l That npm error is almost certainly unrelated to this issue, so it will be best to investigate it separately. It's probably an instance of #1757, which is a path length issue.

[kevintauber]

I also just experienced this issue today. My second push without any changes said everything was up-to-date, but the original changes were successfully updated after a small pseudo change and new push. Here's my original failed attempt errors:

remote: 0 [main] us 0 init_cheap: VirtualAlloc pointer is null, Win32 error 487
remote: AllocationBase 0x0, BaseAddress 0x68570000, RegionSize 0x120000, State 0x10000
remote: D:\Program Files (x86)\Git\bin\sh.exe: *** Couldn't reserve space for cygwin's heap, Win32 error 0

[davidebbo]

We now have a much newer git, and hopefully this should no longer happen.