MinGit can be incompatible with Git for Windows installation of the same bitness #4255
Comments
Yes, this seems to be a likely scenario when using parts of Git for Windows that need a posix-ish environment.
The rebasing method described in https://github.com/git-for-windows/git/wiki/32-bit-issues#adjusting-msys-20dlls-address-range-manually should work around this. Using the busybox based MinGit would theoretically reduce this issue, but it is not currently considered production ready and does not prevent the issue completely IIRC. The general effort to port more scripted parts of git to builtins should also help to slowly reduce this issue in future versions of Git for Windows. When we update our Msys2 runtime to Cygwin 3.4 this issue should theoretically cease to exist completely, but that is currently on hold, because it drops support for i686 and has had a bunch of follow-up issues.
That's good, that avoids imeadiately loading |
|
Even as most of the address-conflict talk is going over my head, I can add that we're able to reproduce this with It's also worth noting that we have client hooks that invoke git.exe directly (via If it's relevant: to get around the fact that VS22 lags substantially behind in Git releases, we also alter PATH early in our hooks such that the system Git is used. I imagine this is not something most folks would need to do. |
It's not what I had in mind when I wrote that part, but it could very well trigger this issue, depending on the shebang.
Right. The Visual studio team has little control over the hooks their users have in their repos. That's why I didn't originally talk about hooks. Invoking git isn't (directly) what's causing this issue. What's causing the issue, is invoking conflicting versions of the various unix tools in Git for Windows (which can happen through git, like for example with hooks). Basically anything in
Sure. We just don't have a reasonable way to make it not crash at the moment.
That might exacerbate the problem in your environment. Assuming your hooks are written as shell scripts their shebang is probably |
When attempting to work around this issue originally, I started with the rebasing approach but the steps as provided were ineffective for the 64-bit installation. I attempted to tweak the specified base address ranges and the parameters passed to My only effective workaround (short of reinstalling Git for Windows with a version identical to the MinGit version embedded in Visual Studio at the time) was to directly copy+overwrite Would it be possible to get an update to that 32-bit wiki with effective steps for a 64-bit installation, if those steps are known? Having a straightforward, scriptable workaround would be extremely helpful given that the proper fixes described above sound like they are still a ways off from being viable. Thank you for the background information and details you've already provided... |
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
|
I investigated this a bit. The root cause of these issues is that the MSYS2 runtime (or for that matter, the Cygwin runtime, from which the MSYS2 runtime is forked) needs to emulate certain POSIX functionality, e.g. the Now, the Cygwin runtime is very careful to version the definition of that Cygwin heap so that even different Cygwin runtimes have a chance to talk to each other. It is quite possible that by rebasing the MSYS2 runtime's custom changes on top of newer Cygwin runtime versions, we miss some steps and inadvertently break that shared memory format versioning. I implemented a pretty big hammer here, which seems to do the job by telling any two different MSYS2 runtime versions that they simply cannot talk to each other. This would most likely address this here ticket, but it might be too big of a hammer (and therefore have unfortunate consequences that I simply haven't had a chance to think through yet). I will let this simmer for a few days, welcoming any insights anybody else might have, and then try to finalize a fix. |
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
|
Thanks all for the contributions to this issue, and to @dscho for getting that patch in. We'll look forward to using it! |
|
@luhansen-ms you're welcome! Could I ask for a favor in return? It would be good to get some confirmation that installing the latest snapshot does indeed fix the problem. Would you mind trying that and reporting back the result? |
|
@dscho Indeed, I haven't been able to reproduce the issue myself, but we'll be syncing tomorrow with our user who can reproduce this consistently. We'll try to validate this with them over the next few days. This was also before the topic of scripted (loading bash/msys) vs built-ins came up in this thread, so I'll give it another try myself. |
|
@dscho, testing with the latest snapshot I am unable to reproduce the bug, and haven't seen any other side effects or odd behaviors. Based on my understanding of the root cause and what you've done here, this looks like it should resolve things for us - thank you. (I should however note that I am currently unable to reproduce the bug with released builds, either... but that's not particularly unusual - in my experience this has been pretty finicky. I've seen pairings of VS 2022 and GfW that will never have trouble because their git versions are synced, and pairings that will usually have trouble because they aren't, but none that will always have trouble.) |
|
Thanks for testing this out @filefull. @dscho Would you expect that only one of the "mismatched versions" would need to be updated to take advantage of the fix? Said another way, either the first or second process to load msys2 could be the version with the fix, since the patched version would have generated the new magic number, and therefore either version would bail out before ever attempting to treat the "first" process as valid msys2/cygwin process to share the heap with? |
@luhansen-ms yes. |
Avoid sharing cygheaps across Cygwin versions It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Avoid sharing cygheaps across Cygwin versions It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Avoid sharing cygheaps across Cygwin versions It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
|
@dscho Is this looking like it would make its way into 2.40? |
|
@luhansen-ms it already made it into v2.39.2... |
When trying to call Cygwin (or for that matter, MSYS2) programs from Git about a "cygheap"](git-for-windows/git#4255) [now](git-for-windows/msys2-runtime#48) allowing basic interactions. While it is still not possible for, say, _is_ now possible for Cygwin's `zstd.exe` in conjuction with Git for Signed-off-by: gitforwindowshelper[bot] <gitforwindowshelper-bot@users.noreply.github.com>
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It frequently leads to problems when trying, say, to call from Git for Windows' Bash into Cygwin's or MSYS2's, merely because sharing that data is pretty finicky. For example, using the Git for Windows that is current at time of writing, trying to call MSYS2's Bash from Git for Windows' Bash fails somewhere in `_cmalloc()`, without any error message, and with the rather misleading exit code 127 (a code which is reserved to indicate that a command was not found). Let's just treat these as completely incompatible with one another, by virtue of using a different `CHILD_INFO_MAGIC` constant. Let's take the msys2-runtime commit as the tell-tale whether two MSYS2 runtime versions are compatible with each other. To support building in the MSYS2-packages repository (where we do not check out the `msys2-runtime` but instead check out Cygwin and apply patches on top), let's accept a hard-coded commit hash as `./configure` option. One consequence is that spawned MSYS processes using a different MSYS2 runtime will not be visible as such to the parent process, i.e. they cannot share any resources such as pseudo terminals. But that's okay, they are simply treated as if they were regular Win32 programs. This should also help the scenario where interactions between two different versions of Git for Windows lead to those infamous `cygheap base mismatch detected` problems mentioned e.g. in git-for-windows/git#4255 Note: We have to use a very rare form of encoding the brackets in the `expr` calls: quadrigraphs (for a thorough explanation, see https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Quadrigraphs.html#Quadrigraphs). This is necessary because it is apparently impossible in `configure.ac` files to encode brackets otherwise. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Summary:
This Issue is in reference to a ticket that was filed with us in Visual Studio. However, we believe the underlying issue to be related to mismatched versions of MinGit/Git for Windows. I haven't been able to reproduce the issue myself, but there is a lot of good info provided by the affected user in the linked ticket that might make more sense here.
Duplicate info from Linked Ticket
Git for Windows includes a component that needs to be pinned in memory at a known location, so that the Unix-like shell that git bash uses can find it. Anything that moves the component – ASLR, rebasing due to address conflicts, etc. – will cause the component to stop working. The specific failure case here seems to be that when you’ve got multiple versions of Git for Windows installed of the same bitness, if that shared component DLL isn’t binary-identical across those versions, the second one that is loaded into memory will be automatically rebased and stop working.
The affected component is msys-2.0.dll, which is present at (for example):
C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\usr\binfor a typical Visual Studio installation; andC:\Program Files\Git\usr\binwithin a typical Git for Windows installation.It seems that this problem may have started occurring when VS 2022 began to include 64-bit git components, at version 17.2.0. The problem is heavily dependent on the version of the Git for Windows installation that is side-by-side with VS 2022; with GfW 2.35.2 against VS 2022 17.2.0, the problem is reproducible. Updating VS 2022 to 17.2.7 moves its internal git installation to 2.37.1 and the problem appears to cease; however, it seems likely that further changes to either git installation could cause the problem to recur.
Any git operation taken from within Visual Studio in this state will fail with a base address mismatch in the MSYS component, with this output:
Questions
I haven't found any existing resources mentioning this happening with 64bit git. Does this sound like an expected scenario? Are there any workarounds besides ensuring that all git processes on the system are compatible versions of Git+msys2?
Setup
defaults?
to the issue you're seeing?
I can ask this in the linked ticket.
Details
Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other
GfW: unknown
MinGit: VS spawns git.exe process directly via Kernel32.CreateProcess
What commands did you run to trigger this issue? If you can provide a
Minimal, Complete, and Verifiable example
this will help us understand the issue.
The text was updated successfully, but these errors were encountered: