Self-sufficient MinGW-based toolchain / packages

[xhochy]

We currently rely on the msys2 channel for providing a MinGW-based toolchain. The main usage of this is currently to build R and its packages. The msys2 channel hasn't been updated for more than 3 years and we are also missing some native dependencies like m2w64-pcre2 in it for the latest R release. To make conda-forge self-sufficient on Windows, we should get rid of the msys2 channel dependency and build up our own set of packages for this.

There are two approaches I could think of, each with its different problems:

1. Build up our own MinGW toolchain from scratch and compile all packages from source. We could also make the compiler as a matrix so that we build some packages in the same recipe for MSVC and MinGW. The disadvantage here is that we will replicate all the work from MSYS2. On the other side this better fits into our infrastructure and would also allow us to apply our own modifications to the source build.
2. Repackage binaries from MSYS2 as the msys2 channel has done in the past. This is the easier approach but we need to think of which packages should be included. All? Only a fixed list?

Approach 2 can probably be scripted mostly using the metadata from MSYS2 and would be the path to go for now to get things up and running.

[xhochy]

For reference, the original source of the msys2 channel is in https://github.com/conda/conda-recipes/tree/master/msys2

[mingwandroid]

Hey @xhochy

Over at AD, we're painfully aware of the issues by our too-old-msys2 packages and we're keen to do something about it. However it is not entirely correct to say that this package suite hasn't been updated in 3 years. Please see the history https://github.com/conda/conda-recipes/commits/master/msys2 and I believe some other team members have updated msys2 packages with recipes located elsewhere (I'd prefer they were all kept here, or all moved, rather than splitting them up like that though).

Since we've now split build and host envs, another very valid approach would be simply to use the MSYS2 GCC patches and build a Linux (or macOS) based GCC cross compiler. Then we'd run the build env as Linux (or WSL) and use e.g. Wine (or when on WSL, native Windows) for testing. MSYS2 is there to provide things like autotools, and other OSes can provide those.

There are other POSIX emulation layers around too, such as midipix. I don't know how appropriate they are. midipix hasn't released much and their dev is done quietly!

If it is decided that CF should stick with MSYS2 (clearly I could be considered biased, but I'm not, I want only whatever's best, and support having as much choice here if possible) then I agree that a binary repackaging might be good for now but that did have some complications. I made conda packages for libalpm and pyalpm (I think) so that the dependencies could be carried across. This worked well, but not perfectly due to some problems with MSYS2 upstream deps (some packages do not declare deps on things installed by the MSYS2 upstream installers). I can try to dig out that work, probably. Then again, the build from source option is really not that bad either.

.. we do this quite a bit. I take the PKGBUILD and patches from MSYS2 directly then just call them. We could also consider compiling pacman for all OSes and run that (or, to be clear, makepkg or makepkg-mingw) from build.sh to do the builds. Luckily, MSYS2, being a split-system (mingw-w64 and msys2), has been engineered to support using cross-compilation (there are projects around on the net now-a-days to both build and install MSYS2 packages on Linux using MSYS2 original recipes or packages). I can dig those out too if you are interested. They work well from what I know.

[xhochy]

Thanks @mingwandroid for the informative reply!

My inference for the three years was having a look at the main packages for the toolchain, I didn't have a look at the git history, sorry for the wrong statement there.

Since we've now split build and host envs, another very valid approach would be simply to use the MSYS2 GCC patches and build a Linux (or macOS) based GCC cross compiler. Then we'd run the build env as Linux (or WSL) and use e.g. Wine (or when on WSL, native Windows) for testing. MSYS2 is there to provide things like autotools, and other OSes can provide those.

This sounds like an attractive approach as that would allow to debug some build issues on Linux. Eventhough I meanwhile got access to a Windows machine, debugging experience is still sadly worse (or worse documented) than on Linux.

There are other POSIX emulation layers around too, such as midipix. I don't know how appropriate they are. midipix hasn't released much and their dev is done quietly!

Personally, I have have also a preference for MSYS2/mingw-w64 as this is the only non-MSVC/clang toolchain on Windows I have seen in several (mostly disjunct) communities in the last years. Using a different toolchain would probably also mean a lot of more work on our side and I don't see any benefits currently (also I'm just unaware of the others).

If it is decided that CF should stick with MSYS2 (clearly I could be considered biased, but I'm not, I want only whatever's best, and support having as much choice here if possible) then I agree that a binary repackaging might be good for now but that did have some complications. I made conda packages for libalpm and pyalpm (I think) so that the dependencies could be carried across. This worked well, but not perfectly due to some problems with MSYS2 upstream deps (some packages do not declare deps on things installed by the MSYS2 upstream installers). I can try to dig out that work, probably. Then again, the build from source option is really not that bad either.

I haven't found anything about the recipes for libalpm either in AnacondaRecipes or conda-forge. Should I look somewhere differently?

Then again, the build from source option is really not that bad either.

This also looks quite interesting. This would enable use to do some small adoptions to the environment while keeping the work on writing the actual build in a single location. If we decide to have our own from-source-builds, this would also be an intermediate step in the migration from binary repackagings.

[mingwandroid]

Great, if CF have a meeting to discuss this I'd like to be involved.

[katietz]

Yeah, great. On discussion I would love to be involved, too

[mingwandroid]

There are some big downsides to mingw-w64/gcc on Windows, mostly handling of AVX. There's an alignment bug which prevents it being used properly. As the person who ported GCC to Windows 64, @katietz will know more!

It is possible to use clang with mingw-w64 these days too. But that's not something either MSYS2 does (they do have clang and are working towards more of the ecosystem being build-able with it), so this is well worth exploring too. It might be nice if we can prototype this idea on behalf of MSYS2. If there are strong advantages.

[xhochy]

cc @isuruf because he worked quite a bit on using clang on Windows and also gave me the massively useful tip to compile r-arrow using clang.

[CJ-Wright]

It would also be great to document this decision and its implications. For instance why can't we use the windows libraries that we usually ship? We build most things (including pcre2 on windows).

[jakirkham]

One other reason we might consider an LLVM based solution is Fortran support recently landed. Though it might be a bit before we see an LLVM release with Fortran support.

[xhochy]

We should also discuss this in the Core meeting at Wednesday 17:00 UTC (@katietz @mingwandroid feel free to join!)

[bilderbuchi]

@jacobwilliams over at the Fortran forum (got no account, only lurking), you were wondering "how to work with Conda Forge to get them to support more recent compilers" (on Windows; e.g. gfortran). I think this here is the relevant issue. 😃

[bilderbuchi]

So, apparently by now gcc and gfortran 13 and 14 builds are available for Windows on conda-forge (congrats and thanks to whoever did this! I could not figure out the source) -- I'm unsure if this means that this can be finally closed?

[owinebar]

So, apparently by now gcc and gfortran 13 and 14 builds are available for Windows on conda-forge (congrats and thanks to whoever did this! I could not figure out the source) -- I'm unsure if this means that this can be finally closed?

Probably, though there is not much documentation on how to use this build chain.
Maybe @xhochy has forgotten he opened this issue.