switch from GNU Make to CMake?

[StefanKarpinski]

I'm creating this issue to discuss the possibility of standardizing our build system on CMake, which seems to be getting significant traction as a saner, cross-platform build system than Make. If we decide to use this to build Julia, we may want to standardize on it as a standard build system for Julia packages that need to compile things too.

[ViralBShah]

#1519 is also related.

[ViralBShah]

Decision is that this is generally a good thing, but not important now since the Windows build works with the current build system.

[ViralBShah]

CMake would be useful if we wanted to build with MSVC and Xcode.

[stevengj]

It would be useful to be able to assume that cmake is available when building packages that wrap around existing source code.

[nolta]

Does Xcode provide cmake?

[ViralBShah]

Xcode does not provide cmake.

[ihnorton]

Small argument in support of CMake: build times, see timings below building llvm-3.2 (all clean source dirs) on core2 duo, 4 GB, Ubuntu 12.04.

./configure Makefile:

time make -j2
...
real    20m45.724s
user    35m4.664s
sys 2m13.084s

CMake Makefile:

time make -j2
...
real    13m31.478s
user    23m6.919s
sys 2m8.340s

CMake Ninja:

time ninja -j2
...
real    13m30.552s
user    23m16.759s
sys 2m1.472s

I used default configure and CMakeLists options (so maybe slightly different options...). Ninja doesn't make a big difference in cold build time on Linux, but small changes or no-op builds are much faster (no-op: ~0s vs 3.5s). And If Julia ever supports MSVC, the improvement is enormous: escape from msbuild.

Some relevant discussions of Ninja speed:
http://news.ycombinator.com/item?id=3896622
http://public.kitware.com/pipermail/cmake-developers/2012-February/003318.html
https://plus.google.com/108996039294665965197/posts/SfhrFAhRyyd

[stevengj]

Scratch my earlier comment, by the way. I've concluded that cmake is useless for packages because packages can't assume that Mac or Windows systems have compilers (hence you have to distribute binaries for those systems).

[vharavy]

I am not an advocate of CMake, but in my opinion current Julia build system demands serious attention. I spent about 4 hours trying to built 64-bit Julia on Windows. The only working method for now is using cross-compile.

Currently I am investigating the possibility of using Julia at work. I found Julia to be a way ahead of R and MATLAB in the matter of performance but there are some moments:

1. At the moment the only way to build Julia on Windows is using MinGW and MSYS. I find it a little bit annoying. It would be very useful to have ability to build Julia using other compilers, notable Microsoft Visual C++ and Intel C++ Composer. The latter includes math library and Intel Math Kernel Library which contains implementation of BLAS, LAPACK, SparseBLAS, Fourier Transforms, PARDISO, Vector Math Library etc. At the company I work we have an Intel C++ Composer and I believe it will be possibly for us to provide pre-built 64-bit binaries (and 32-bit if required) of Julia for Windows. Also, Intel C++ Composer for Linux is free for open-source projects so there is a possibility to get similar builds for Linux.
2. Currently installation of Julia includes by default a lot of functionality. It would be nice to have the ability to select whether to install it or not - i.e. move stuff like big integers, sparse matrices, Fourier transforms into packages and exclude them from the build. Thus Julia will build faster, start faster, consume less disk space and memory.

P.S. I would glad to participate in the development (especially in first clause).

[vtjnash]

.1a. There's already the option of downloading pre-built binaries from https://code.google.com/p/julialang/downloads/list
However, building with the intel compiler could give a nice speed boost.

.1b. The MKL can already be used with MinGW/MSYS/llvm/gcc compiled julia. I've made this change before locally. However, we don't have permission to distribute MKL so the default install uses OpenBLAS.

.2. While it would speed up the first build, it makes no difference to the speed of future builds to exclude these other libraries. Excluding them would save < 20 MB of disk space and have a negligible impact on startup speed (the current slow startup is unrelated to the number of libraries -- there's a different issue open for fixing that). Thus, there really isn't a particularly good reason to exclude these items.

PS. The windows build instructions are at https://github.com/JuliaLang/julia/blob/master/README.windows.md for both native and cross builds. Others have found that they are correct. However, Windows is very slow at some critical operations and thus will take much longer than 4 hours for the initial build. I believe the problem lies with how much better optimized linux file access is compared to Windows and since compiling inherently requires reading and writing a large number of files. Switching to cmake will probably help reduce compile time, but only somewhat.

[vharavy]

1a. There is no 64-bit pre-built binaries for Windows.,
1b. My company have the rights to build and distribute software with Intel C++ Composer.

P.S. I have read those Windows build instructions very carefully and still I have got some issues. First one is make produces several errors like the following:

make: *** No rule to make target `/d/Temporary/usr/bin', needed by `release'.  Stop.

It is cured by creating these folders manually. The second one appears when building some of the dependencies. It seems that somewhere on the way of compiling make messes with destination directory and produces output files in

/d/d/Temporary/usr/lib

while expecting them to be in

/d/Temporary/usr/lib

I can provide you with some details if you like. But my post was about build system in general. I believe it is too complicated for building Windows builds. E.g. necessity to install MinGW and MSYS (around 800Mb in total) seems strange. My opinion is build system of any software should have as least requirements (or dependencies) as possible.

[vtjnash]

1a. Not yet (v0.1 didn't build on windows 64), but there will be starting with v0.2 in a month or so. In the meantime, the 32-bit binary is identical in many respects.

1b. That would probably be helpful. We've talked about doing that for all of the builds, but nobody has had sufficient time or interest to make it happen so far. It is probably easiest to get the linux version working first. IIUC, the right to distribute programs compiled with icc is separate from the right to distribute MKL. Switching a pre-built 32-bit julia binary to MKL should be possible by editing the blas name in build_h.jl file and deleting the sys.ji cache file.

P.S. I think I found and fixed the cause of the first error. It is probably more appropriate to open a new issue for the second item (although it may be fixed by the first?).

[JeffBezanson]

At this time we believe we will not switch to cmake.

[twadleigh]

Is this an issue closed only for lack of urgency/volunteers? If so, I might be willing to do some work in toward this end.

[tkelman]

I think it's definitely worth revisiting this, see some recent discussion in #8937 (comment) (and this is the more correct place to discuss it, thanks for finding this issue!)

Especially now that we have the libgit2 dependency which requires cmake to be installed, and we have partially functional MSVC builds, I think this would be a good transition to investigate further. I've done a very small amount of work in this direction but would be interested in expanding it. Perhaps starting with something small like configuration probes for common build issues that we've encountered?

[ViralBShah]

Certainly worth revisiting. Time to start learning cmake.

[maleadt]

CMake also includes "External Project" functionality which should allow managing deps. Although that would be quite a big porting effort...

[stevengj]

Given that it's probably not practical to port all of the dependencies to cmake, what will we gain by switching Julia itself to cmake?

[tkelman]

A few things:

1. a separate "configure" step to detect system configuration and requirements ahead of time, rather than finding problems midway through a long build - doing this in cmake would be more modern and maintainable than autoconf or a custom script
2. sane non-hacky support for MSVC
3. other generators like ninja which are much better than make at incremental builds
   4) out-of-tree builds (I really want this) mostly works with makefiles now

[gitfoxi]

The problem with cmake is that cmake hard to build and has version issues. Like if you want to build cmake on an old system you need old cmake and hope what you actually wanted to build doesn't use new features. Or you can use old cmake to build incrementally newer cmake ...

Why not SCons? SCons is easy assuming you have Python 2.4 or newer.

If you're not happy with some of SCons tradeoffs they're nice to provide a comparison with lots of alternative build systems.

[tkelman]

How old of a system are you talking about? Yes cmake has grown new features over time, but it also makes a point of declaring your minimum version requirements right up front. We haven't had any complaints yet about cmake versions not being new enough for libgit2 on master, so I think all the versions on systems people are commonly building Julia on have been good enough so far. Modern cmake has a fairly simple bootstrap system to compile it from source if you're somehow on a distribution where you can't get new enough versions from your package manager or binary installers. It's not at all hard to build relative to the things we're using in Julia, and most people shouldn't need to build it anyway.

We already need cmake. Many of our dependencies have cmake build systems already written, so much of the fragile boilerplate repetition we currently have in deps/Makefile should be possible to simplify quite a bit using ExternalProject. Unless you want to convert all of the dependencies to scons there's no reason to be different.

[twadleigh]

@tkelman, I was thinking I might set some time aside to help work on moving this forward over the holiday. I'll send you an e-mail.

[vtjnash]

discussion can continue on the PR (#11754), if necessary. however, i don't think we really need to revisit this. the slow build times and msvc issues are as much a function of our dependencies (if not more), and switching julia to cmake wouldn't address that.

[tkelman]

msvc on gmake is a hack that doesn't work at all in msys2 (they broke that) so that would really be the main motivation for switching. agreed though that the deps are the messy part of this.

[habemus-papadum]

one small win for having a cmake build it that is some ide's have very good support for project configuration (clion and qt creator mainly, and but also in a slightly less integrated way, eclipse/xcode/visual studio (clion/qt creator make an effort at grokking cmake, and allowing users to make intelligent changes to both the source and cmake build files, the others are supported by cmake but there are limits to the scope of changes that are convenient to make in a cmake generated IDE project))

As an example, since llvm uses cmake, a great way to start grokking how it works it to clone it, and open the CMakelists.txt in clion and start from various examples and hop "backwards" into deeper parts of the core system. Not only do you have robust/template aware navigation within the entire code base, but you can easily run/debug examples, or just start hacking including adding new files, new libraries, etc. The net experience is a much lower barrier to entry for new developers who want to grok the code base. (nb. clion can be slow on llvm due to its size and project complexity though I find usable on a high end workstation)

Along the same lines, aside from cmake, a minimal but useful improvement would be to make sure that the current Makefile based system generates a compile_commands.json file (cmake builds provide this easily, but there are many other systems (e.g.Bear) that also achieve this for other scenarios like Makefile projects). This opens the possibility for using various clang based tools (e.g. rtags, FB's nuclide, maybe about 8 others) to help initially grokking the code base.

Basic point it that using a main stream build system opens the possibility for new developers to use existing tools accelerate the speed at which they can grok julia internals.

Should these comments be on the PR(#11754)?

[jmigual]

So, CMake has gone a long way since this issue was opened almost 11 years ago. I think right now CMake is quite a mature setup (still with some minor quirks) that is supported in most of the IDEs. Would it make sense to reopen this discussion about problems and solutions?

[StefanKarpinski]

You can comment on a closed issue—absent actual evidence that there's a problem, we're not going to reopen.

What are some actual problems that you (or anyone) has with Julia's use of make? Make is the worst build tool, except for all the others.

[jmigual]

Indeed, I am revisiting this closed issue because it was concluded 7 years ago, and I believe it's worth reevaluating in light of changes and advancements since then. I don't think it's fair to be penalized for trying to assess what has evolved and how we could improve. Here are some areas where Make falls short and where CMake could potentially offer improvements:

• Windows Compatibility: Currently, using Make on Windows requires installing an entire environment that emulates a Unix setup, such as Cygwin or MSYS2. This introduces an additional layer of complexity and potential sources of errors for Windows users.
• Compiler Support: CMake provides extensive support for a variety of compilers, including MSVC and clang-cl, offering more flexibility and options for users.
• Existing Dependency: It's worth noting that CMake is already a build dependency of Julia.
• Management of External Dependencies: CMake simplifies the management of external dependencies through features like ExternalDependency and FetchContent, making the build process more efficient and manageable.

In a properly set up CMake environment, a user unfamiliar with the project could build and install Julia as follows, with dependencies included:

git clone https://github.com/JuliaLang/julia.git
cmake -B build -S julia
cmake --build build
cmake --install build

I believe these points substantiate the consideration of CMake as an alternative to Make for Julia’s build system. I look forward to further discussion on this matter.

[StefanKarpinski]

I don't think it's fair to be penalized for trying to assess what has evolved and how we could improve.

Nobody is being penalized, I just don't want to reopen an issue for no reason.

Windows Compatibility: Currently, using Make on Windows requires installing an entire environment that emulates a Unix setup, such as Cygwin or MSYS2. This introduces an additional layer of complexity and potential sources of errors for Windows users.

WSL seems like a pretty good option for those who want to develop Julia on Windows systems. But sure, point taken, being able to build Julia natively without MSYS2 would be nice. That said, WSL, Cygwin and MSYS2 do all exist.

Compiler Support: CMake provides extensive support for a variety of compilers, including MSVC and clang-cl, offering more flexibility and options for users.

I don't think we're interested in supporting building Julia with other compilers. We tried supporting building Julia with MSVC and it was not a success. It never worked, it was a lot of work to even try to support, and unless we waste a lot of CI resources constantly testing it, it will just bitrot. MSVC is different enough from GCC and Clang that even if we could support it, I fear we'd constantly be telling people not to build Julia that way because the end result isn't officially supported and will inevitably have bugs.

Existing Dependency: It's worth noting that CMake is already a build dependency of Julia.

Only if you're building all dependencies from source, which there's not much reason to do these days.

Management of External Dependencies: CMake simplifies the management of external dependencies through features like ExternalDependency and FetchContent, making the build process more efficient and manageable.

I guess, but we have a system that works very well already and switching to this would be a big process for little benefit. If we didn't have a working system for this, sure, but we do.

In a properly set up CMake environment, a user unfamiliar with the project could build and install Julia as follows, with dependencies included:

git clone https://github.com/JuliaLang/julia.git
cmake -B build -S julia
cmake --build build
cmake --install build

Ok, but you can already build and install Julia just by doing

git clone https://github.com/JuliaLang/julia.git
make -C julia install

The final question of course is: Who would do this? Are you willing to spearhead changing the build system to CMake? If you want to do that and you come up with a much better build system, I don't think anyone would object, but I think that the devs are pretty ok with using make. This seems like a lot of work to go from having a working build system to having a (different) working build system. This is open source, so you can't just argue that something would be better and expect someone else to do it. Someone who really wants Julia to use CMake has to make it happen—you can't just make an argument that it would be better and expect others to do it for you because you made a good case. You have to either do it or convince someone to do it.

[jmigual]

I see, you make a good case for the other points. So, the biggest "issue" of the current build system is the dependency on MSYS2.

The final question of course is: Who would do this? Are you willing to spearhead changing the build system to CMake? If you want to do that and you come up with a much better build system, I don't think anyone would object, but I think that the devs are pretty ok with using make. This seems like a lot of work to go from having a working build system to having a (different) working build system. This is open source, so you can't just argue that something would be better and expect someone else to do it. Someone who really wants Julia to use CMake has to make it happen—you can't just make an argument that it would be better and expect others to do it for you because you made a good case. You have to either do it or convince someone to do it.

Regarding this, I would suggest having both build systems but keeping CMake as unofficially supported. This means that the julia developers don't work on the CMake build system, but that PRs are accepted. I've seen multiple projects go through these steps (mixxdj or C++ boost libraries being two examples).

Right now, there's a fork that started adding CMake support (https://github.com/Neumann-A/julia/tree/add_cmake) although I was not able to build it. So, we could start from there.

[PallHaraldsson]

I would suggest having both build systems but keeping CMake as unofficially supported.

That seems like the worst of both worlds, duplication, to have two systems, unless it's a temporary thing until you migrate, and you actually plan to do that.

I thought you meant Boost has both, but I find no Makefile there, only CMake support. The unofficial CMake can you made to work in a fork, but I don't strictly oppose it here.

Also I have no say in this, I can sort of read Makefiles, but don't really want to edit them (here), nor need to. They are ugly and CMake seems maybe more readable, but as you wrote it doesn't (yet) work for Julia...

Would it be valuable to only make CMake for Windows work? I suppose such a file could be simpler. And the other Makefile could maybe refer to it for other platforms, needing both..., but not at first?

[jmigual]

That seems like the worst of both worlds, duplication, to have two systems, unless it's a temporary thing until you migrate, and you actually plan to do that.

I think that even if temporary, it can still take a very long time. While, I'm confident on CMake, I know nothing on the build requirements of julia so the build transition would require me to read and understand lots of Makefiles.

I thought you meant Boost has booth, but I find no Makefile there, only CMake support.

Boost always used a build system called b2 similar to autotools which was designed to build only boost. That system is still there and is the currently officially supported system. They've been trying to move to CMake for more than 5 years and right now it's on a usable state. In their case, the move to CMake is more important than for Julia, as, being Boost a library, it needs to be easy to incorporate with other projects and CMake helps a lot in that sense.

The unofficial CMake can you made to work in a fork, but I don't strictly oppose it here.

Good, even if it's on a fork, the best way would be to avoid deleting any existing build file to guarantee that the original system still works. The fork that I pointed deleted some files so it cannot be directly added to julia right now.

Also I have no say in this, I can sort of read Makefiles, but don't really want to edit them (here), nor need to. They are ugly and CMake seems maybe more readable, but as you wrote it doesn't (yet) work for Julia...

Would it be valuable to only make CMake for Windows work? I suppose such a file could be simpler. And the other Makefile could maybe refer to it for other platforms, needing both..., but not at first?

This could be a start yes. On the other side, if you write your CMake files properly from the beginning, it is quite easy to port to other platforms. The stages I see are:

1. Julia builds on Windows with CMake
2. Tests build and pass on Windows with CMake
3. Julia builds all platforms with CMake
4. Tests build and pass on all platforms with CMake

Two questions I have are:

1. If we had a PR adding CMake. At which stage (1-4) would we accept a PR adding it?
2. Would it be acceptable to use a package manager with the CMake build such as vcpkg or CPM?

[ViralBShah]

We would start with an experiment that shows 2 (additional package managers seem reasonable if they are official on that platform). But maintaining 2 build systems is going to be tough, because all the maintainers, CI scripts etc. have to deal with it. So it can only be accepted when 4 is done, and it has to have all the bells and whistles we have today with full source builds of the deps (many of which have their own bespoke build systems), binary builder builds, and so on.

There is also a commitment to maintaining the new build system going forward. So there's the risk that even when ready, if it doesn't look right, or we don't feel we'll have enough committers feeling comfortable to maintain, we may not be able to switch over.

I would say that it is not impossible, but the risks of switching are high, while the payoff is medium.

If I had to pick a way forward, achieving 2 in an external repo (like the other example) and seeing how committers feel would be a good milestone. It would be nice to hear what @vtjnash @gbaraldi @vchuravy feel.

[vchuravy]

I once tried doing this. I didn't enjoy it... I also recommend looking how other large projects use CMake. LLVM our biggest dependency essentially has written a new CMake-dialect to support their needs.

When I have fever dreams about this I tend to want to try out meson.

Would it be acceptable to use a package manager with the CMake build

I am very hesitant on this. We need not just provide an environment for building Julia, but make sure that all the binaries used are ABI compatible. We achieve this right now by standardizing the build and using Yggdrasil to cross-compile libraries for windows.

[vtjnash]

We have enough cmake dependencies already, that option 5 instead is probably acceptable: convert only some part, but convert it completely. An example of this might be to move the llvm.mk and libuv.mk files to be cmake subprojects of a singular cmake project in deps (with only shims remaining in the deps/Makefile)